Text_Classification/classification_imdb.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text classification with Keras\n",
    "\n",
    "This notebook uses keras layers to show examples of training a text classifier model. Models with different combinations of Attention, LSTM and GRU have been shown. Glove and Fastext embeddings are used to initialize the word embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('2.2.4',\n",
       " '1.11.0',\n",
       " '3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)]')"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras, tensorflow, sys\n",
    "keras.__version__, tensorflow.__version__, sys.version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install two required Attention Packages\n",
    "\n",
    "pip install keras_multi_head\n",
    "\n",
    "Link to the package -> https://pypi.org/project/keras-multi-head/\n",
    "\n",
    "pip install keras_self_attention\n",
    "\n",
    "Link to the package -> https://pypi.org/project/keras-self-attention/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "C:\\Users\\Deep Learning 3033\\AppData\\Local\\conda\\conda\\envs\\tensorflow\\lib\\site-packages\\sklearn\\cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "# import required packages\n",
    "import sys\n",
    "import warnings\n",
    "\n",
    "if not sys.warnoptions:\n",
    "    warnings.simplefilter(\"ignore\")\n",
    "\n",
    "import keras\n",
    "from keras.layers import CuDNNLSTM,CuDNNGRU, BatchNormalization, Dense, Dropout, Activation, Embedding, Input, Concatenate\n",
    "from keras.layers import Bidirectional,CuDNNGRU,SpatialDropout1D, GlobalAveragePooling1D, GlobalMaxPooling1D\n",
    "from keras.models import Model\n",
    "from keras.optimizers import Adam\n",
    "import tensorflow as tf\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.utils.np_utils import to_categorical\n",
    "from keras_multi_head import MultiHeadAttention\n",
    "from keras_self_attention import ScaledDotProductAttention, SeqSelfAttention\n",
    "\n",
    "from sklearn.metrics import confusion_matrix,f1_score, precision_score, recall_score, roc_auc_score, accuracy_score\n",
    "from sklearn.cross_validation import train_test_split\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import re\n",
    "from glob import glob\n",
    "\n",
    "import math"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"1\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IMDB Dataset\n",
    "\n",
    "The task here is to categorize the incoming comment as positive or negative. \n",
    "\n",
    "Find the dataset at the link -> http://ai.stanford.edu/~amaas/data/sentiment/\n",
    "\n",
    "load_imdb_dataset() function is used to load the imdb data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_imdb_dataset():\n",
    "\n",
    "    # Load the dataset\n",
    "    train = pd.DataFrame(columns=[\"text\", \"positive\"])\n",
    "    test = pd.DataFrame(columns=[\"text\", \"positive\"])\n",
    "    ctr = 0\n",
    "    cte = 0\n",
    "    for fil in ['train/', 'test/']:\n",
    "        for cls in ['pos', 'neg']:\n",
    "            dset_path = \"./\" + fil + cls\n",
    "            for fname in sorted(os.listdir(dset_path)):\n",
    "                if fname.endswith('.txt'):\n",
    "                    with open(os.path.join(dset_path, fname), encoding=\"utf8\") as f:\n",
    "                        if fil == 'train/':\n",
    "                            train.loc[ctr] = (f.read(), int(cls == \"pos\"))\n",
    "                            ctr+=1\n",
    "                        else:\n",
    "                            test.loc[cte] = (f.read(), int(cls == \"pos\"))\n",
    "                            cte+=1\n",
    "                            \n",
    "    return train, test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Number of sample and there distribution in test and train sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train data shape (25000, 2)\n",
      "Test data shape (25000, 2)\n"
     ]
    }
   ],
   "source": [
    "train, test = load_imdb_dataset()\n",
    "\n",
    "print (\"Train data shape\", train.shape)\n",
    "print (\"Test data shape\", test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train data class distbn 1    12500\n",
      "0    12500\n",
      "Name: positive, dtype: int64\n",
      "Test data class distbn 1    12500\n",
      "0    12500\n",
      "Name: positive, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(\"Train data class distbn\", train.positive.value_counts())\n",
    "print(\"Test data class distbn\", test.positive.value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Few sample comments from both test and train sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>positive</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Bromwell High is a cartoon comedy. It ran at t...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Homelessness (or Houselessness as George Carli...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Brilliant over-acting by Lesley Ann Warren. Be...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>This is easily the most underrated film inn th...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>This is not the typical Mel Brooks film. It wa...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text positive\n",
       "0  Bromwell High is a cartoon comedy. It ran at t...        1\n",
       "1  Homelessness (or Houselessness as George Carli...        1\n",
       "2  Brilliant over-acting by Lesley Ann Warren. Be...        1\n",
       "3  This is easily the most underrated film inn th...        1\n",
       "4  This is not the typical Mel Brooks film. It wa...        1"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>positive</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>I went and saw this movie last night after bei...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Actor turned director Bill Paxton follows up h...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>As a recreational golfer with some knowledge o...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>I saw this film in a sneak preview, and it is ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Bill Paxton has taken the true story of the 19...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text positive\n",
       "0  I went and saw this movie last night after bei...        1\n",
       "1  Actor turned director Bill Paxton follows up h...        1\n",
       "2  As a recreational golfer with some knowledge o...        1\n",
       "3  I saw this film in a sneak preview, and it is ...        1\n",
       "4  Bill Paxton has taken the true story of the 19...        1"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Sequence length distribution:\n",
      "\n",
      "count    25000.000000\n",
      "mean       233.787200\n",
      "std        173.733032\n",
      "min         10.000000\n",
      "25%        127.000000\n",
      "50%        174.000000\n",
      "75%        284.000000\n",
      "max       2470.000000\n",
      "dtype: float64\n",
      "\n",
      "\n",
      "Test Sequence length distribution:\n",
      "\n",
      "count    25000.000000\n",
      "mean       228.526680\n",
      "std        168.883693\n",
      "min          4.000000\n",
      "25%        126.000000\n",
      "50%        172.000000\n",
      "75%        277.000000\n",
      "max       2278.000000\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# Average number of words per review \n",
    "tr_l = [len(x.split()) for x in train.text]\n",
    "te_l = [len(x.split()) for x in test.text]\n",
    "print(\"Train Sequence length distribution:\\n\")\n",
    "print(pd.Series(tr_l).describe())\n",
    "print(\"\\n\\nTest Sequence length distribution:\\n\")\n",
    "print(pd.Series(te_l).describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocab size 88582\n"
     ]
    }
   ],
   "source": [
    "# Number of unique words by finding the length of dictionary of words mapped with unique tokens (integers)\n",
    "tokenizer = Tokenizer()\n",
    "tokenizer.fit_on_texts(list(train.text))\n",
    "print(\"Vocab size\", len(tokenizer.word_counts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the sentence length to be the mean number of words per sentence in the train set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "embed_size = 300 \n",
    "\n",
    "# mean number of words per sentence in the train set is taken as maximum sentence length.\n",
    "max_sent_len = int(np.percentile(tr_l, 50)) \n",
    "\n",
    "num_words = len(tokenizer.word_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenize and pad the text sequences\n",
    "\n",
    "Tokenize -> change the word to there integer ids\n",
    "\n",
    "Pad -> Trim or pad with zeros to make all sentences of same length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Converte sentence text to list of token represented sentences, required for training\n",
    "X = tokenizer.texts_to_sequences(train.text)\n",
    "X = pad_sequences(X, maxlen=max_sent_len)\n",
    "\n",
    "x_test = tokenizer.texts_to_sequences(test.text)\n",
    "x_test = pad_sequences(x_test, maxlen=max_sent_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the validataion data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((22500, 174), (2500, 174))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Split into train and validation data\n",
    "x_train, x_val, y_train, y_val = train_test_split(X, train.positive, test_size=0.1, random_state=3)\n",
    "x_train.shape, x_val.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions to load different Embedding files.\n",
    "\n",
    "Load the embedding file and find the mean and standard deviation vectors ot the word vectors. Than for all the words in the vocab initialize the corresponding word vector from the loaded embedded file. For the words for which wordvecs cannot be found in the embedding file, initialize them with a random normal distribution with the above found mean and standard deviation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_glove(word_index):\n",
    "    EMBEDDING_FILE = '../embeddings/glove.840B.300d/glove.840B.300d.txt'\n",
    "    def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\n",
    "    embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE, encoding=\"utf8\"))\n",
    "\n",
    "    all_embs = np.stack(embeddings_index.values())\n",
    "    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n",
    "    embed_size = all_embs.shape[1]\n",
    "\n",
    "    # word_index = tokenizer.word_index\n",
    "    nb_words = min(num_words, len(word_index))\n",
    "    embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n",
    "    for word, i in word_index.items():\n",
    "        if i >= num_words: continue\n",
    "        embedding_vector = embeddings_index.get(word)\n",
    "        if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n",
    "            \n",
    "    return embedding_matrix \n",
    "\n",
    "def load_google_news(word_index):\n",
    "    EMBEDDING_FILE = '../embeddings/GoogleNews-vectors-negative300/GoogleNews-vectors-negative300.bin'\n",
    "    def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\n",
    "    embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE, encoding=\"utf8\"))\n",
    "\n",
    "    all_embs = np.stack(embeddings_index.values())\n",
    "    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n",
    "    embed_size = all_embs.shape[1]\n",
    "\n",
    "    # word_index = tokenizer.word_index\n",
    "    nb_words = min(num_words, len(word_index))\n",
    "    embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n",
    "    for word, i in word_index.items():\n",
    "        if i >= num_words: continue\n",
    "        embedding_vector = embeddings_index.get(word)\n",
    "        if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n",
    "            \n",
    "    return embedding_matrix \n",
    "    \n",
    "def load_fasttext(word_index):    \n",
    "    EMBEDDING_FILE = '../embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec'\n",
    "    def get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\n",
    "    embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE, encoding=\"utf8\") if len(o)>100)\n",
    "\n",
    "    all_embs = np.stack(embeddings_index.values())\n",
    "    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n",
    "    embed_size = all_embs.shape[1]\n",
    "\n",
    "    # word_index = tokenizer.word_index\n",
    "    nb_words = min(num_words, len(word_index))\n",
    "    embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\n",
    "    for word, i in word_index.items():\n",
    "        if i >= num_words: continue\n",
    "        embedding_vector = embeddings_index.get(word)\n",
    "        if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n",
    "\n",
    "    return embedding_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_index = tokenizer.word_index\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train Function\n",
    "\n",
    "Trains the model. The parameters decides what model to be trained and on what type of embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(embd=0, train_embd=True, bilstm=True, attn=0, avg_pool=False):\n",
    "## embd -> 0 if not to use pretrained embedding \n",
    "#       -> 1 if Glove pretrained vectors is to be used\n",
    "#       -> 2 if FastText pretrained vectors is to be used\n",
    "#       -> 3 if mean of Glove and FastText pretrained vectors is to be used.\n",
    "#\n",
    "## train_embd should be either of True or False depending upon Embedding layer is to be fine tuned or not.\n",
    "#\n",
    "## bilstm -> False if Bidirectional GRU is to be used\n",
    "#         -> True if Bidirectional LSTM is to be used.\n",
    "#\n",
    "## attn -> 0 No attention\n",
    "#       -> 1 Sequence self attention\n",
    "#       -> 2 Multi-head attention\n",
    "#       -> 3 Use both Sequence and multi head attention and concatenate the outputs.\n",
    "#\n",
    "## avg_pool Set to True to concatenate average_pool along with max pool output in the network.\n",
    "    \n",
    "    pred_avg = []\n",
    "    real = list(test.positive)\n",
    "    \n",
    "    # Performing cross validation of 5 for result consistency.\n",
    "    for cv in range(5):\n",
    "        embedding_layer = None\n",
    "\n",
    "        if (embd==0):\n",
    "            embedding_layer = Embedding(input_dim=num_words, output_dim=embed_size, input_length=max_sent_len,\n",
    "                                        trainable=train_embd)\n",
    "\n",
    "        elif (embd==1):\n",
    "            embedding_layer = Embedding(input_dim=num_words, output_dim=embed_size, input_length=max_sent_len,\n",
    "                                        trainable=train_embd, weights=[load_glove(word_index)])\n",
    "        elif (embd==2):\n",
    "            embedding_layer = Embedding(input_dim=num_words, output_dim=embed_size, input_length=max_sent_len,\n",
    "                                        trainable=train_embd, weights=[load_fasttext(word_index)])\n",
    "        elif (embd==3):    \n",
    "            embedding_layer = Embedding(input_dim=num_words, output_dim=embed_size, input_length=max_sent_len,\n",
    "                                        trainable=train_embd, \n",
    "                                        weights=[np.mean([load_glove(word_index), load_fasttext(word_index)], axis = 0)])\n",
    "\n",
    "\n",
    "        sequence_input = Input(shape=(max_sent_len,), dtype='int32')\n",
    "        embedded_sequences = embedding_layer(sequence_input)\n",
    "        x = SpatialDropout1D(rate=0.2)(embedded_sequences)\n",
    "        # Spatial drop out will drop a whole 1d word vector of incoming sentence, \n",
    "        # rather than dropping random units from any of the word vectors. \n",
    "\n",
    "        if bilstm:\n",
    "            x = Bidirectional(CuDNNLSTM(units=64, return_sequences=True), merge_mode='concat')(x)\n",
    "        else:\n",
    "            x = Bidirectional(CuDNNGRU(units=64, return_sequences=True), merge_mode='concat')(x)\n",
    "\n",
    "\n",
    "        if attn==1:\n",
    "            x = SeqSelfAttention()(x)\n",
    "\n",
    "        if attn==2:\n",
    "            x = MultiHeadAttention(head_num=4)(x)\n",
    "\n",
    "        if attn==3:\n",
    "            x1 = SeqSelfAttention()(x)\n",
    "            x2 = MultiHeadAttention(head_num=4)(x)\n",
    "            if avg_pool:\n",
    "                xg1 = GlobalMaxPooling1D()(x1)\n",
    "                xa1 = GlobalAveragePooling1D()(x1)\n",
    "                xg2 = GlobalMaxPooling1D()(x2)\n",
    "                xa2 = GlobalAveragePooling1D()(x2)\n",
    "                x = Concatenate()([xg1, xa1, xg2, xa2])\n",
    "            else:\n",
    "                x1 = GlobalMaxPooling1D()(x1)\n",
    "                x2 = GlobalMaxPooling1D()(x2)\n",
    "                x = Concatenate()([x1, x2])\n",
    "        else:\n",
    "            if avg_pool:\n",
    "                xg = GlobalMaxPooling1D()(x)\n",
    "                xa = GlobalAveragePooling1D()(x)\n",
    "                x = Concatenate()([xg, xa])\n",
    "            else:\n",
    "                x = GlobalMaxPooling1D()(x)\n",
    "\n",
    "\n",
    "        x = Dense(units=16, activation=\"relu\", kernel_initializer=\"glorot_normal\")(x)\n",
    "        x = BatchNormalization()(x)\n",
    "        x = Dropout(rate=0.4)(x)\n",
    "\n",
    "        pred = Dense(units=1, activation=\"sigmoid\", kernel_initializer=\"glorot_normal\")(x)\n",
    "        model = Model(sequence_input, pred)\n",
    "\n",
    "        model.compile(loss=\"binary_crossentropy\", optimizer=Adam(5e-5),metrics=['accuracy'])\n",
    "\n",
    "        model.fit(x=x_train, y=y_train, validation_data=(x_val, y_val), epochs=30, batch_size=128, \n",
    "                  shuffle=True, verbose=0)\n",
    "\n",
    "        pred = model.predict(x=x_test)\n",
    "        pred = pred > 0.5\n",
    "        pred = [int(p[0]) for p in pred]\n",
    "        pred_avg.append(pred)\n",
    "        print(\"Model:\", cv, \", Accuracy_score:\", accuracy_score(real, pred))\n",
    "    \n",
    "    pred = np.mean(pred_avg, axis=0)\n",
    "    pred = pred > 0.5\n",
    "    pred = [int(p) for p in pred]\n",
    "    print(\"Confusion Matrix:\\n\", confusion_matrix(real, pred))\n",
    "    print(\"f1_score:\",f1_score(real, pred), \"precision_score:\",precision_score(real, pred),\n",
    "          \"recall_score:\",recall_score(real, pred), \"accuracy_score:\",accuracy_score(real, pred))\n",
    "    return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results for different models trained on the IMDB dataset\n",
    "\n",
    "The results suggests that ->\n",
    "1. Use of LSTM or GRU must be experimented for a given dataset. Normally LSTM works better in tasks where long term dependency in sequence matters for making the classification.\n",
    "\n",
    "2. Choice of embedding will also depend on the task and one may choose to use combination of embeddings. Note here Concatenation of embedding have not been shown in this notebook, but one may combine embeddings in that way. But generally, after concatenating multiple embeddings, the results detoriate. \n",
    "\n",
    "3. Fine tuning of embeddings usually helps when there is enough data and data has fairly good number of examples for words which were not present in embedding vocublary. \n",
    "\n",
    "4. Using Attention layer have given improvements of above 1%. Sequence self attention seems to work better for this problem but combination of attentions may produce even better results, as has been shown below.\n",
    "\n",
    "5. Using concatenation of global average pooled features and global max-pooled features may generally produce better results. In this case concatenating average pooled features has given an improvement of another 0.5-1%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: 0 , Accuracy_score: 0.8382\n",
      "Model: 1 , Accuracy_score: 0.83828\n",
      "Model: 2 , Accuracy_score: 0.8282\n",
      "Model: 3 , Accuracy_score: 0.83884\n",
      "Model: 4 , Accuracy_score: 0.8394\n",
      "Confusion Matrix:\n",
      " [[10870  1630]\n",
      " [ 2232 10268]]\n",
      "f1_score: 0.8417083367489139 precision_score: 0.863002185241217 recall_score: 0.82144 accuracy_score: 0.84552\n"
     ]
    }
   ],
   "source": [
    "# base model - embedding randomly intitialized, Using LSTM, no attention.\n",
    "train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: 0 , Accuracy_score: 0.86064\n",
      "Model: 1 , Accuracy_score: 0.86112\n",
      "Model: 2 , Accuracy_score: 0.85936\n",
      "Model: 3 , Accuracy_score: 0.86092\n",
      "Model: 4 , Accuracy_score: 0.8628\n",
      "Confusion Matrix:\n",
      " [[10965  1535]\n",
      " [ 1597 10903]]\n",
      "f1_score: 0.8744085331622424 precision_score: 0.8765878758642869 recall_score: 0.87224 accuracy_score: 0.87472\n"
     ]
    }
   ],
   "source": [
    "# base model - embedding randomly intitialized, Using GRU, no attention.\n",
    "train(bilstm=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: 0 , Accuracy_score: 0.86192\n",
      "Model: 1 , Accuracy_score: 0.86748\n",
      "Model: 2 , Accuracy_score: 0.86356\n",
      "Model: 3 , Accuracy_score: 0.86416\n",
      "Model: 4 , Accuracy_score: 0.86636\n",
      "Confusion Matrix:\n",
      " [[10911  1589]\n",
      " [ 1628 10872]]\n",
      "f1_score: 0.8711189455550659 precision_score: 0.8724821442901853 recall_score: 0.86976 accuracy_score: 0.87132\n"
     ]
    }
   ],
   "source": [
    "# Using embedding Glove, No fine tune, Using GRU, no attention.\n",
    "train(embd=1, train_embd=False, bilstm=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: 0 , Accuracy_score: 0.873\n",
      "Model: 1 , Accuracy_score: 0.87424\n",
      "Model: 2 , Accuracy_score: 0.87404\n",
      "Model: 3 , Accuracy_score: 0.87176\n",
      "Model: 4 , Accuracy_score: 0.87428\n",
      "Confusion Matrix:\n",
      " [[10950  1550]\n",
      " [ 1459 11041]]\n",
      "f1_score: 0.8800765214618788 precision_score: 0.8768961956953379 recall_score: 0.88328 accuracy_score: 0.87964\n"
     ]
    }
   ],
   "source": [
    "# Using embedding Glove, No fine tune, Using LSTM, no attention.\n",
    "train(embd=1, train_embd=False, bilstm=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: 0 , Accuracy_score: 0.87604\n",
      "Model: 1 , Accuracy_score: 0.87756\n",
      "Model: 2 , Accuracy_score: 0.87604\n",
      "Model: 3 , Accuracy_score: 0.8814\n",
      "Model: 4 , Accuracy_score: 0.8768\n",
      "Confusion Matrix:\n",
      " [[11098  1402]\n",
      " [ 1434 11066]]\n",
      "f1_score: 0.8864146107016981 precision_score: 0.8875521334616618 recall_score: 0.88528 accuracy_score: 0.88656\n"
     ]
    }
   ],
   "source": [
    "# Using embedding Glove, fine tuning, Using GRU, no attention.\n",
    "train(embd=1, train_embd=True, bilstm=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: 0 , Accuracy_score: 0.88644\n",
      "Model: 1 , Accuracy_score: 0.888\n",
      "Model: 2 , Accuracy_score: 0.8856\n",
      "Model: 3 , Accuracy_score: 0.88808\n",
      "Model: 4 , Accuracy_score: 0.88376\n",
      "Confusion Matrix:\n",
      " [[11111  1389]\n",
      " [ 1288 11212]]\n",
      "f1_score: 0.8933508625154376 precision_score: 0.889770653122768 recall_score: 0.89696 accuracy_score: 0.89292\n"
     ]
    }
   ],
   "source": [
    "# Using embedding Glove, fine tuning, Using LSTM, no attention.\n",
    "train(embd=1, train_embd=True, bilstm=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: 0 , Accuracy_score: 0.85296\n",
      "Model: 1 , Accuracy_score: 0.85712\n",
      "Model: 2 , Accuracy_score: 0.8652\n",
      "Model: 3 , Accuracy_score: 0.8664\n",
      "Model: 4 , Accuracy_score: 0.86064\n",
      "Confusion Matrix:\n",
      " [[10383  2117]\n",
      " [ 1192 11308]]\n",
      "f1_score: 0.8723625843780135 precision_score: 0.8423091247672253 recall_score: 0.90464 accuracy_score: 0.86764\n"
     ]
    }
   ],
   "source": [
    "# Using embedding Fastext, No fine tune, Using GRU, no attention.\n",
    "train(embd=2, train_embd=False, bilstm=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: 0 , Accuracy_score: 0.86636\n",
      "Model: 1 , Accuracy_score: 0.863\n",
      "Model: 2 , Accuracy_score: 0.86236\n",
      "Model: 3 , Accuracy_score: 0.86484\n",
      "Model: 4 , Accuracy_score: 0.86224\n",
      "Confusion Matrix:\n",
      " [[10818  1682]\n",
      " [ 1540 10960]]\n",
      "f1_score: 0.871847903905815 precision_score: 0.866951431735485 recall_score: 0.8768 accuracy_score: 0.87112\n"
     ]
    }
   ],
   "source": [
    "# Using embedding as mean of Glove and Fastext, No fine tune, Using GRU, no attention.\n",
    "train(embd=3, train_embd=False, bilstm=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: 0 , Accuracy_score: 0.87948\n",
      "Model: 1 , Accuracy_score: 0.88204\n",
      "Model: 2 , Accuracy_score: 0.8778\n",
      "Model: 3 , Accuracy_score: 0.87932\n",
      "Model: 4 , Accuracy_score: 0.88364\n",
      "Confusion Matrix:\n",
      " [[10997  1503]\n",
      " [ 1403 11097]]\n",
      "f1_score: 0.884223107569721 precision_score: 0.8807142857142857 recall_score: 0.88776 accuracy_score: 0.88376\n"
     ]
    }
   ],
   "source": [
    "# Using embedding Glove, No fine tune, Using GRU, Sequence Self-Attention.\n",
    "train(embd=1, train_embd=False, bilstm=False, attn = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: 0 , Accuracy_score: 0.87672\n",
      "Model: 1 , Accuracy_score: 0.8812\n",
      "Model: 2 , Accuracy_score: 0.8842\n",
      "Model: 3 , Accuracy_score: 0.87912\n",
      "Model: 4 , Accuracy_score: 0.87592\n",
      "Confusion Matrix:\n",
      " [[10918  1582]\n",
      " [ 1304 11196]]\n",
      "f1_score: 0.8858295751246142 precision_score: 0.8761934575050868 recall_score: 0.89568 accuracy_score: 0.88456\n"
     ]
    }
   ],
   "source": [
    "# Using embedding Glove, No fine tune, Using GRU, Multi-head Attention.\n",
    "train(embd=1, train_embd=False, bilstm=False, attn = 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: 0 , Accuracy_score: 0.873\n",
      "Model: 1 , Accuracy_score: 0.88292\n",
      "Model: 2 , Accuracy_score: 0.88172\n",
      "Model: 3 , Accuracy_score: 0.88024\n",
      "Model: 4 , Accuracy_score: 0.8806\n",
      "Confusion Matrix:\n",
      " [[10771  1729]\n",
      " [ 1179 11321]]\n",
      "f1_score: 0.886183953033268 precision_score: 0.8675095785440613 recall_score: 0.90568 accuracy_score: 0.88368\n"
     ]
    }
   ],
   "source": [
    "# Using embedding Glove, No fine tune, Using GRU, Concat of self and Multi head Attentions.\n",
    "train(embd=1, train_embd=False, bilstm=False, attn = 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: 0 , Accuracy_score: 0.8928\n",
      "Model: 1 , Accuracy_score: 0.89204\n",
      "Model: 2 , Accuracy_score: 0.8928\n",
      "Model: 3 , Accuracy_score: 0.8928\n",
      "Model: 4 , Accuracy_score: 0.88996\n",
      "Confusion Matrix:\n",
      " [[11235  1265]\n",
      " [ 1279 11221]]\n",
      "f1_score: 0.8981829824701834 precision_score: 0.8986865289123819 recall_score: 0.89768 accuracy_score: 0.89824\n"
     ]
    }
   ],
   "source": [
    "# Using embedding Glove , No fine tune, Using GRU, Sequence Self-Attention, concat of average and max pool.\n",
    "train(embd=1, train_embd=True, bilstm=False, attn = 1, avg_pool=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: 0 , Accuracy_score: 0.88364\n",
      "Model: 1 , Accuracy_score: 0.88896\n",
      "Model: 2 , Accuracy_score: 0.88716\n",
      "Model: 3 , Accuracy_score: 0.88964\n",
      "Model: 4 , Accuracy_score: 0.88512\n",
      "Confusion Matrix:\n",
      " [[11192  1308]\n",
      " [ 1350 11150]]\n",
      "f1_score: 0.8935010818174534 precision_score: 0.8950072242735592 recall_score: 0.892 accuracy_score: 0.89368\n"
     ]
    }
   ],
   "source": [
    "# Using embedding Glove , No fine tune, Using GRU, Multi-head Attention, concat of average and max pool.\n",
    "train(embd=1, train_embd=True, bilstm=False, attn = 2, avg_pool=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: 0 , Accuracy_score: 0.88268\n",
      "Model: 1 , Accuracy_score: 0.88324\n",
      "Model: 2 , Accuracy_score: 0.87912\n",
      "Model: 3 , Accuracy_score: 0.88304\n",
      "Model: 4 , Accuracy_score: 0.88188\n",
      "Confusion Matrix:\n",
      " [[10923  1577]\n",
      " [ 1300 11200]]\n",
      "f1_score: 0.8861811132650236 precision_score: 0.8765750958754012 recall_score: 0.896 accuracy_score: 0.88492\n"
     ]
    }
   ],
   "source": [
    "# Using embedding Glove, No fine tune, Using GRU, Concat of self and Multi head attention, concat of avg and max pool.\n",
    "train(embd=1, train_embd=False, bilstm=False, attn = 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}