Notebooks/LSTM_bidirectional_IMDB_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bidirectional LSTM on the IMDB sentiment classification task.\n",
    "### Dr. Tirthajyoti Sarkar, Fremont, CA\n",
    "\n",
    "IMDB dataset: https://keras.io/datasets/#imdb-movie-reviews-sentiment-classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import numpy as np\n",
    "\n",
    "from keras.preprocessing import sequence\n",
    "from keras.optimizers import Adam\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Embedding, LSTM, Bidirectional\n",
    "from keras.callbacks import ModelCheckpoint, LearningRateScheduler, ReduceLROnPlateau\n",
    "from keras.datasets import imdb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many features you will consider and how long embedding is to be used\n",
    "Note, these choices will determine the size of the parameter space of your neural net."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_features = 2000\n",
    "# cut texts after this number of words\n",
    "# (among top max_features most common words)\n",
    "maxlen = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the data\n",
    "\n",
    "Keras IMDB data sometimes gives trouble while loading. <br>\n",
    "See this Stakoverflow thread for possible resolution,\n",
    "\n",
    "https://stackoverflow.com/questions/55890813/how-to-fix-object-arrays-cannot-be-loaded-when-allow-pickle-false-for-imdb-loa/56062555"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data...\n",
      "25000 train sequences\n",
      "25000 test sequences\n"
     ]
    }
   ],
   "source": [
    "print('Loading data...')\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
    "print(len(x_train), 'train sequences')\n",
    "print(len(x_test), 'test sequences')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pad sequences (samples x time)\n",
      "x_train shape: (25000, 100)\n",
      "x_test shape: (25000, 100)\n"
     ]
    }
   ],
   "source": [
    "print('Pad sequences (samples x time)')\n",
    "x_train = sequence.pad_sequences(x_train, maxlen=maxlen)\n",
    "x_test = sequence.pad_sequences(x_test, maxlen=maxlen)\n",
    "print('x_train shape:', x_train.shape)\n",
    "print('x_test shape:', x_test.shape)\n",
    "y_train = np.array(y_train)\n",
    "y_test = np.array(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build your model\n",
    "Feel free to experiment with number of neurons and add more LSTM layers. But the irreducible error bar of the dataset can be reched pretty easily even with a simple network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From c:\\users\\tirth\\docume~1\\personal\\datasc~2\\python~1\\tf-gpu\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From c:\\users\\tirth\\docume~1\\personal\\datasc~2\\python~1\\tf-gpu\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(Embedding(max_features, 128, input_length=maxlen))\n",
    "model.add(Bidirectional(LSTM(64)))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimizer and compilation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "opti_ = Adam(lr=0.0025)\n",
    "model.compile(optimizer=opti_, loss='binary_crossentropy', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 100, 128)          256000    \n",
      "_________________________________________________________________\n",
      "bidirectional_1 (Bidirection (None, 128)               98816     \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 129       \n",
      "=================================================================\n",
      "Total params: 354,945\n",
      "Trainable params: 354,945\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fix a batch size, number of epochs to train, and off you go..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 64\n",
    "epochs = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From c:\\users\\tirth\\docume~1\\personal\\datasc~2\\python~1\\tf-gpu\\lib\\site-packages\\tensorflow\\python\\ops\\math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "25000/25000 [==============================] - 151s 6ms/step - loss: 0.4437 - acc: 0.7908 - val_loss: 0.3670 - val_acc: 0.8370\n",
      "Epoch 2/20\n",
      "25000/25000 [==============================] - 150s 6ms/step - loss: 0.3345 - acc: 0.8566 - val_loss: 0.3542 - val_acc: 0.8458\n",
      "Epoch 3/20\n",
      "25000/25000 [==============================] - 145s 6ms/step - loss: 0.3104 - acc: 0.8681 - val_loss: 0.3353 - val_acc: 0.8557\n",
      "Epoch 4/20\n",
      "25000/25000 [==============================] - 144s 6ms/step - loss: 0.2571 - acc: 0.8942 - val_loss: 0.3547 - val_acc: 0.8527\n",
      "Epoch 5/20\n",
      "25000/25000 [==============================] - 145s 6ms/step - loss: 0.2231 - acc: 0.9090 - val_loss: 0.3777 - val_acc: 0.8408\n",
      "Epoch 6/20\n",
      "25000/25000 [==============================] - 147s 6ms/step - loss: 0.1970 - acc: 0.9228 - val_loss: 0.3859 - val_acc: 0.8465\n",
      "Epoch 7/20\n",
      "25000/25000 [==============================] - 146s 6ms/step - loss: 0.1649 - acc: 0.9373 - val_loss: 0.4530 - val_acc: 0.8497\n",
      "Epoch 8/20\n",
      "25000/25000 [==============================] - 146s 6ms/step - loss: 0.1449 - acc: 0.9444 - val_loss: 0.4873 - val_acc: 0.8438\n",
      "Epoch 9/20\n",
      "25000/25000 [==============================] - 147s 6ms/step - loss: 0.1155 - acc: 0.9566 - val_loss: 0.5251 - val_acc: 0.8359\n",
      "Epoch 10/20\n",
      "25000/25000 [==============================] - 146s 6ms/step - loss: 0.0960 - acc: 0.9658 - val_loss: 0.5556 - val_acc: 0.8355\n",
      "Epoch 11/20\n",
      "25000/25000 [==============================] - 145s 6ms/step - loss: 0.0748 - acc: 0.9730 - val_loss: 0.6368 - val_acc: 0.8260\n",
      "Epoch 12/20\n",
      "25000/25000 [==============================] - 147s 6ms/step - loss: 0.0680 - acc: 0.9770 - val_loss: 0.6851 - val_acc: 0.8398\n",
      "Epoch 13/20\n",
      "25000/25000 [==============================] - 146s 6ms/step - loss: 0.0618 - acc: 0.9788 - val_loss: 0.7714 - val_acc: 0.8341\n",
      "Epoch 14/20\n",
      "25000/25000 [==============================] - 149s 6ms/step - loss: 0.0413 - acc: 0.9871 - val_loss: 0.8070 - val_acc: 0.8380\n",
      "Epoch 15/20\n",
      "25000/25000 [==============================] - 148s 6ms/step - loss: 0.0324 - acc: 0.9896 - val_loss: 0.9204 - val_acc: 0.8324\n",
      "Epoch 16/20\n",
      "25000/25000 [==============================] - 148s 6ms/step - loss: 0.0435 - acc: 0.9854 - val_loss: 0.8847 - val_acc: 0.8338\n",
      "Epoch 17/20\n",
      "25000/25000 [==============================] - 152s 6ms/step - loss: 0.0248 - acc: 0.9929 - val_loss: 0.9444 - val_acc: 0.8340\n",
      "Epoch 18/20\n",
      "25000/25000 [==============================] - 151s 6ms/step - loss: 0.0254 - acc: 0.9924 - val_loss: 0.9751 - val_acc: 0.8355\n",
      "Epoch 19/20\n",
      "25000/25000 [==============================] - 149s 6ms/step - loss: 0.0316 - acc: 0.9892 - val_loss: 0.9276 - val_acc: 0.8228\n",
      "Epoch 20/20\n",
      "25000/25000 [==============================] - 149s 6ms/step - loss: 0.0341 - acc: 0.9883 - val_loss: 0.9594 - val_acc: 0.8263\n"
     ]
    }
   ],
   "source": [
    "t1=time()\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          validation_data=[x_test, y_test])\n",
    "t2=time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20 took total 49.265 minutes.\n"
     ]
    }
   ],
   "source": [
    "t_delta = round((t2-t1)/60,3)\n",
    "print(f\"{epochs} took total {t_delta} minutes.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some fancy things to try - `LearningRateScheduler` and `ReduceLROnPlateau` Callbacks\n",
    "These are very useful callbacks to dynamically adjust the learning rate but they don't seem to impact the performance for this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lr_schedule(epoch):\n",
    "    \"\"\"Learning Rate Schedule\n",
    "\n",
    "    Learning rate is scheduled to be reduced after 80, 120, 160, 180 epochs.\n",
    "    Called automatically every epoch as part of callbacks during training.\n",
    "\n",
    "    # Arguments\n",
    "        epoch (int): The number of epochs\n",
    "\n",
    "    # Returns\n",
    "        lr (float32): learning rate\n",
    "    \"\"\"\n",
    "    lr = 0.005\n",
    "    if epoch >= 3:\n",
    "        lr *= 0.5\n",
    "    if epoch >= 7:\n",
    "        lr *= 0.25\n",
    "    if epoch >= 11:\n",
    "        lr *= 0.5\n",
    "    if epoch >= 16:\n",
    "        lr *= 0.5\n",
    "        \n",
    "    print('Learning rate: ', lr)\n",
    "    return lr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_scheduler = LearningRateScheduler(lr_schedule)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_reducer = ReduceLROnPlateau(factor=np.sqrt(0.1),\n",
    "                               cooldown=0,\n",
    "                               patience=5,\n",
    "                               min_lr=0.5e-6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "callbacks = [lr_reducer, lr_scheduler]\n",
    "#callbacks = [lr_scheduler]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Embedding(max_features, 128, input_length=maxlen))\n",
    "model.add(Bidirectional(LSTM(64)))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learning rate:  0.005\n"
     ]
    }
   ],
   "source": [
    "opti_ = Adam(lr=lr_schedule(0))\n",
    "model.compile(optimizer=opti_, loss='binary_crossentropy', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/20\n",
      "Learning rate:  0.005\n",
      "25000/25000 [==============================] - 152s 6ms/step - loss: 0.5266 - acc: 0.7340 - val_loss: 0.4281 - val_acc: 0.8036\n",
      "Epoch 2/20\n",
      "Learning rate:  0.005\n",
      "25000/25000 [==============================] - 143s 6ms/step - loss: 0.3857 - acc: 0.8278 - val_loss: 0.3792 - val_acc: 0.8277\n",
      "Epoch 3/20\n",
      "Learning rate:  0.005\n",
      "25000/25000 [==============================] - 145s 6ms/step - loss: 0.3255 - acc: 0.8610 - val_loss: 0.3572 - val_acc: 0.8411\n",
      "Epoch 4/20\n",
      "Learning rate:  0.0025\n",
      "25000/25000 [==============================] - 143s 6ms/step - loss: 0.2665 - acc: 0.8888 - val_loss: 0.3616 - val_acc: 0.8477\n",
      "Epoch 5/20\n",
      "Learning rate:  0.0025\n",
      "25000/25000 [==============================] - 145s 6ms/step - loss: 0.2340 - acc: 0.9050 - val_loss: 0.3770 - val_acc: 0.8463\n",
      "Epoch 6/20\n",
      "Learning rate:  0.0025\n",
      "25000/25000 [==============================] - 146s 6ms/step - loss: 0.2087 - acc: 0.9166 - val_loss: 0.4063 - val_acc: 0.8430\n",
      "Epoch 7/20\n",
      "Learning rate:  0.0025\n",
      "25000/25000 [==============================] - 146s 6ms/step - loss: 0.1831 - acc: 0.9282 - val_loss: 0.4244 - val_acc: 0.8403\n",
      "Epoch 8/20\n",
      "Learning rate:  0.000625\n",
      "25000/25000 [==============================] - 144s 6ms/step - loss: 0.1307 - acc: 0.9528 - val_loss: 0.4712 - val_acc: 0.8398\n",
      "Epoch 9/20\n",
      "Learning rate:  0.000625\n",
      "25000/25000 [==============================] - 143s 6ms/step - loss: 0.1087 - acc: 0.9618 - val_loss: 0.5328 - val_acc: 0.8384\n",
      "Epoch 10/20\n",
      "Learning rate:  0.000625\n",
      "25000/25000 [==============================] - 144s 6ms/step - loss: 0.0935 - acc: 0.9683 - val_loss: 0.5648 - val_acc: 0.8372\n",
      "Epoch 11/20\n",
      "Learning rate:  0.000625\n",
      "25000/25000 [==============================] - 144s 6ms/step - loss: 0.0816 - acc: 0.9734 - val_loss: 0.6201 - val_acc: 0.8334\n",
      "Epoch 12/20\n",
      "Learning rate:  0.0003125\n",
      "25000/25000 [==============================] - 145s 6ms/step - loss: 0.0662 - acc: 0.9798 - val_loss: 0.6597 - val_acc: 0.8327\n",
      "Epoch 13/20\n",
      "Learning rate:  0.0003125\n",
      "25000/25000 [==============================] - 144s 6ms/step - loss: 0.0592 - acc: 0.9830 - val_loss: 0.6892 - val_acc: 0.8324\n",
      "Epoch 14/20\n",
      "Learning rate:  0.0003125\n",
      "25000/25000 [==============================] - 144s 6ms/step - loss: 0.0532 - acc: 0.9852 - val_loss: 0.7253 - val_acc: 0.8318\n",
      "Epoch 15/20\n",
      "Learning rate:  0.0003125\n",
      "25000/25000 [==============================] - 144s 6ms/step - loss: 0.0482 - acc: 0.9866 - val_loss: 0.7542 - val_acc: 0.8300\n",
      "Epoch 16/20\n",
      "Learning rate:  0.0003125\n",
      "25000/25000 [==============================] - 144s 6ms/step - loss: 0.0437 - acc: 0.9881 - val_loss: 0.7774 - val_acc: 0.8282\n",
      "Epoch 17/20\n",
      "Learning rate:  0.00015625\n",
      "25000/25000 [==============================] - 144s 6ms/step - loss: 0.0377 - acc: 0.9906 - val_loss: 0.8038 - val_acc: 0.8292\n",
      "Epoch 18/20\n",
      "Learning rate:  0.00015625\n",
      "25000/25000 [==============================] - 143s 6ms/step - loss: 0.0353 - acc: 0.9910 - val_loss: 0.8214 - val_acc: 0.8289\n",
      "Epoch 19/20\n",
      "Learning rate:  0.00015625\n",
      "25000/25000 [==============================] - 144s 6ms/step - loss: 0.0327 - acc: 0.9919 - val_loss: 0.8452 - val_acc: 0.8289\n",
      "Epoch 20/20\n",
      "Learning rate:  0.00015625\n",
      "25000/25000 [==============================] - 143s 6ms/step - loss: 0.0301 - acc: 0.9934 - val_loss: 0.8690 - val_acc: 0.8285\n"
     ]
    }
   ],
   "source": [
    "t1=time()\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          validation_data=[x_test, y_test],\n",
    "          callbacks=callbacks)\n",
    "t2=time()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": ""
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}