ml/keras-test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LSTM with Keras\n",
    "\n",
    "Based on a [Kaggle solution](https://www.kaggle.com/kredy10/simple-lstm-for-text-classification) by kredy10. See also tutorial by [Shivam Bansal](https://www.kaggle.com/shivamb/beginners-guide-to-text-generation-using-lstms) and on [Bag of Words](https://www.kaggle.com/c/word2vec-nlp-tutorial#part-1-for-beginners-bag-of-words).\n",
    "\n",
    "Dataset from [usableprivacy.org](https://usableprivacy.org/data/):\n",
    "\n",
    "> [Identifying the Provision of Choices in Privacy Policy Text](https://usableprivacy.org/static/files/ksathyendra_acl_2017.pdf). Kanthashree Mysore Sathyendra, Shomir Wilson, Florian Schaub, Sebastian Zimmeck, and Norman Sadeh. Conference on Empirical Methods in Natural Language Processing (EMNLP), Copenhagen, Denmark, Sep 2017"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get this code to run, you need a Python data science environment. There are lots of guides around to get this set up, for example [this one](https://forum.schoolofdata.ch/t/interpreting-git-with-python/259) (schoolofdata.ch). If using Anaconda, install [keras](https://anaconda.org/conda-forge/keras) from conda-forge, as well as pandas, numpy, seaborn, matplotlib and sklearn, before continuing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "from keras.models import Model\n",
    "from keras.layers import LSTM, Activation, Dense, Dropout, Input, Embedding\n",
    "from keras.optimizers import RMSprop\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing import sequence\n",
    "from keras.utils import to_categorical\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read in data from our sample file, and tweak it a bit (it's just a test) to create some lower case text content without funny characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'we have engaged third party companies to process information about our users on our behalf to help us improve our sites and find new members these companies will combine our anonymized data about how you visit our websites with information that they have collected from outside our sites and provide it to us and only us so that we can optimize your experience on fool com across devices and based on your interests we will not provide the third party service providers with your name address email address or telephone number if you would like more information about the information we are receiving and your choices with respect to that data please visit the daa opt out program the evidon opt out program to learn how to opt out of having your information collected in browsers for ad serving purposes download the evidon ad control application available for ios and android to learn how to opt out of having your information collected in mobile apps and visit the network advertising initiative s opt out page '"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('OptOutTypeDataset.csv',delimiter=',',encoding='latin-1')\n",
    "df.a2.replace(\"BR\", \"FI\", inplace = True)\n",
    "df.a2.replace(\"XX\", \"TH\", inplace = True)\n",
    "# Combine text columns\n",
    "df['lowtext'] = df.fulltext.str.lower() # + \" \" + df.text.str.lower()\n",
    "# Strip HTML tags\n",
    "df['lowtext'] = df.lowtext.str.replace(\"<[^>]*>\", \"\", regex=True)\n",
    "# Remove all punctuation\n",
    "df['lowtext'] = df.lowtext.str.replace('[^0-9a-zA-Z]+', ' ', regex=True)\n",
    "# Show the first line\n",
    "df.lowtext[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 102 entries, 0 to 101\n",
      "Data columns (total 15 columns):\n",
      "id          102 non-null object\n",
      "a1          91 non-null object\n",
      "a2          102 non-null object\n",
      "a3          101 non-null object\n",
      "a4          102 non-null object\n",
      "a5          90 non-null object\n",
      "a6          102 non-null object\n",
      "a7          101 non-null object\n",
      "a8          102 non-null object\n",
      "text        102 non-null object\n",
      "button      102 non-null object\n",
      "url         102 non-null object\n",
      "fulltext    102 non-null object\n",
      "html        102 non-null object\n",
      "lowtext     102 non-null object\n",
      "dtypes: object(15)\n",
      "memory usage: 12.0+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>a1</th>\n",
       "      <th>a2</th>\n",
       "      <th>a3</th>\n",
       "      <th>a4</th>\n",
       "      <th>a5</th>\n",
       "      <th>a6</th>\n",
       "      <th>a7</th>\n",
       "      <th>a8</th>\n",
       "      <th>text</th>\n",
       "      <th>button</th>\n",
       "      <th>url</th>\n",
       "      <th>fulltext</th>\n",
       "      <th>html</th>\n",
       "      <th>lowtext</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1259,9,4,0</td>\n",
       "      <td>TH</td>\n",
       "      <td>TH</td>\n",
       "      <td>TH</td>\n",
       "      <td>TH</td>\n",
       "      <td>SH</td>\n",
       "      <td>AD</td>\n",
       "      <td>AD</td>\n",
       "      <td>AD</td>\n",
       "      <td>visit the &lt;a href=\"http://www.aboutads.info/ch...</td>\n",
       "      <td>DAA opt-out program</td>\n",
       "      <td>http://www.aboutads.info/choices/</td>\n",
       "      <td>We have engaged third-party companies to proce...</td>\n",
       "      <td>1259_fool.com.html</td>\n",
       "      <td>we have engaged third party companies to proce...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1259,9,4,1</td>\n",
       "      <td>FI</td>\n",
       "      <td>TH</td>\n",
       "      <td>BR</td>\n",
       "      <td>TH</td>\n",
       "      <td>SH</td>\n",
       "      <td>AD</td>\n",
       "      <td>AD</td>\n",
       "      <td>AD</td>\n",
       "      <td>visit the &lt;a href=\"http://www.aboutads.info/ch...</td>\n",
       "      <td>Evidon opt-out program</td>\n",
       "      <td>http://choices.ghosteryenterprise.com/global-o...</td>\n",
       "      <td>We have engaged third-party companies to proce...</td>\n",
       "      <td>1259_fool.com.html</td>\n",
       "      <td>we have engaged third party companies to proce...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1259,9,4,2</td>\n",
       "      <td>TH</td>\n",
       "      <td>TH</td>\n",
       "      <td>TH</td>\n",
       "      <td>TH</td>\n",
       "      <td>AD</td>\n",
       "      <td>AD</td>\n",
       "      <td>AD</td>\n",
       "      <td>AD</td>\n",
       "      <td>visit the &lt;a href=\"http://www.aboutads.info/ch...</td>\n",
       "      <td>Network Advertising Initiative's opt out page</td>\n",
       "      <td>http://www.networkadvertising.org/choices/</td>\n",
       "      <td>We have engaged third-party companies to proce...</td>\n",
       "      <td>1259_fool.com.html</td>\n",
       "      <td>we have engaged third party companies to proce...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>133,10,2,0</td>\n",
       "      <td>FI, TH</td>\n",
       "      <td>FI</td>\n",
       "      <td>FI</td>\n",
       "      <td>FI</td>\n",
       "      <td>AD, SH</td>\n",
       "      <td>CM, SH</td>\n",
       "      <td>AD, SH</td>\n",
       "      <td>AD, SH</td>\n",
       "      <td>If you prefer not to receive e-mail marketing ...</td>\n",
       "      <td>click here</td>\n",
       "      <td>https://subscription.timeinc.com/storefront/pr...</td>\n",
       "      <td>Privacy Options   If you prefer not to receive...</td>\n",
       "      <td>133_fortune.com.html</td>\n",
       "      <td>privacy options if you prefer not to receive t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>82,13,2,0</td>\n",
       "      <td>FI</td>\n",
       "      <td>FI</td>\n",
       "      <td>FI</td>\n",
       "      <td>FI</td>\n",
       "      <td>CM</td>\n",
       "      <td>CM</td>\n",
       "      <td>CM</td>\n",
       "      <td>CM</td>\n",
       "      <td>If you do not want to receive such email messa...</td>\n",
       "      <td>legal@sheknows.com</td>\n",
       "      <td>mailto:legal@sheknows.com?subject=No</td>\n",
       "      <td>By providing Company your email address, you c...</td>\n",
       "      <td>82_drinksmixer.com.html</td>\n",
       "      <td>by providing company your email address you co...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           id      a1  a2  a3  a4      a5      a6      a7      a8  \\\n",
       "0  1259,9,4,0      TH  TH  TH  TH      SH      AD      AD      AD   \n",
       "1  1259,9,4,1      FI  TH  BR  TH      SH      AD      AD      AD   \n",
       "2  1259,9,4,2      TH  TH  TH  TH      AD      AD      AD      AD   \n",
       "3  133,10,2,0  FI, TH  FI  FI  FI  AD, SH  CM, SH  AD, SH  AD, SH   \n",
       "4   82,13,2,0      FI  FI  FI  FI      CM      CM      CM      CM   \n",
       "\n",
       "                                                text  \\\n",
       "0  visit the <a href=\"http://www.aboutads.info/ch...   \n",
       "1  visit the <a href=\"http://www.aboutads.info/ch...   \n",
       "2  visit the <a href=\"http://www.aboutads.info/ch...   \n",
       "3  If you prefer not to receive e-mail marketing ...   \n",
       "4  If you do not want to receive such email messa...   \n",
       "\n",
       "                                          button  \\\n",
       "0                            DAA opt-out program   \n",
       "1                         Evidon opt-out program   \n",
       "2  Network Advertising Initiative's opt out page   \n",
       "3                                     click here   \n",
       "4                             legal@sheknows.com   \n",
       "\n",
       "                                                 url  \\\n",
       "0                  http://www.aboutads.info/choices/   \n",
       "1  http://choices.ghosteryenterprise.com/global-o...   \n",
       "2         http://www.networkadvertising.org/choices/   \n",
       "3  https://subscription.timeinc.com/storefront/pr...   \n",
       "4               mailto:legal@sheknows.com?subject=No   \n",
       "\n",
       "                                            fulltext                     html  \\\n",
       "0  We have engaged third-party companies to proce...       1259_fool.com.html   \n",
       "1  We have engaged third-party companies to proce...       1259_fool.com.html   \n",
       "2  We have engaged third-party companies to proce...       1259_fool.com.html   \n",
       "3  Privacy Options   If you prefer not to receive...     133_fortune.com.html   \n",
       "4  By providing Company your email address, you c...  82_drinksmixer.com.html   \n",
       "\n",
       "                                             lowtext  \n",
       "0  we have engaged third party companies to proce...  \n",
       "1  we have engaged third party companies to proce...  \n",
       "2  we have engaged third party companies to proce...  \n",
       "3  privacy options if you prefer not to receive t...  \n",
       "4  by providing company your email address you co...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f8300a37f60>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAADelJREFUeJzt3X+s3fVdx/HnCzoGzCGUXlilYNHUbUQd6AlBSfzBD8PiXOscZIvM61ZT/3Bzy/xB3R9u8UfCIjrmtixpylhZtgFhQslcVFLBaTTILSPjd8BKWENHL1ACOAVb3/5xv43X0vaeW/ieb+nn+Uhuzvl+7/ec77vJzX32e77ne26qCklSu44aegBJ0rAMgSQ1zhBIUuMMgSQ1zhBIUuMMgSQ1zhBIUuMMgSQ1zhBIUuOW9PnkSU4ENgI/ChTwAeBh4AZgJfAYcFlV7TrY8yxbtqxWrlzZ56iSdMTZunXrU1U1tdB26fMjJpJsAv6xqjYmOQY4HvgY8ExVXZlkPXBSVV1xsOcZjUY1MzPT25ySdCRKsrWqRgtt19tLQ0lOAH4GuAagql6qqmeB1cCmbrNNwJq+ZpAkLazPcwQ/BMwC1yb5VpKNSd4AnFpVOwC621N6nEGStIA+Q7AE+Ang81V1DvAfwPpxH5xkXZKZJDOzs7N9zShJzeszBNuB7VV1Z7d8E3NheDLJcoDuduf+HlxVG6pqVFWjqakFz3VIkg5RbyGoqu8C30ny5m7VhcADwK3AdLduGtjc1wySpIX1+vZR4EPAl7t3DG0D3s9cfG5MshZ4HLi05xkkSQfRawiq6h5gf29durDP/UqSxueVxZLUOEMgSY3r+xzBYeMnf++6oUfQYWbrn/3a0CNIhwWPCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhq3pM8nT/IY8DywB9hdVaMkS4EbgJXAY8BlVbWrzzkkSQc2iSOCn6+qs6tq1C2vB7ZU1SpgS7csSRrIEC8NrQY2dfc3AWsGmEGS1Ok7BAX8XZKtSdZ1606tqh0A3e0p+3tgknVJZpLMzM7O9jymJLWr13MEwPlV9USSU4Dbkjw07gOragOwAWA0GlVfA0pS63o9IqiqJ7rbncDNwLnAk0mWA3S3O/ucQZJ0cL2FIMkbkrxx733gF4D7gFuB6W6zaWBzXzNIkhbW50tDpwI3J9m7n69U1d8kuQu4Mcla4HHg0h5nkCQtoLcQVNU24G37Wf80cGFf+5UkLY5XFktS4wyBJDXOEEhS4wyBJDXOEEhS4wyBJDXOEEhS4wyBJDXOEEhS4wyBJDXOEEhS4wyBJDXOEEhS4wyBJDXOEEhS4wyBJDWu7z9eL2kBj//Rjw09gg5DZ/zhvRPbl0cEktQ4QyBJjTMEktQ4QyBJjTMEktQ4QyBJjTMEktQ4QyBJjTMEktS43kOQ5Ogk30ry9W75zCR3JnkkyQ1Jjul7BknSgU3iiODDwIPzlj8JfKqqVgG7gLUTmEGSdAC9hiDJCuAXgY3dcoALgJu6TTYBa/qcQZJ0cH0fEVwN/D7wP93yycCzVbW7W94OnNbzDJKkg+gtBEneAeysqq3zV+9n0zrA49clmUkyMzs728uMkqR+jwjOB96Z5DHgeuZeEroaODHJ3o+/XgE8sb8HV9WGqhpV1WhqaqrHMSWpbb2FoKr+oKpWVNVK4D3A31fVrwK3A+/uNpsGNvc1gyRpYUNcR3AF8NEkjzJ3zuCaAWaQJHUm8hfKquoO4I7u/jbg3EnsV5K0MK8slqTGGQJJapwhkKTGGQJJapwhkKTGGQJJapwhkKTGGQJJapwhkKTGGQJJapwhkKTGGQJJapwhkKTGGQJJapwhkKTGGQJJapwhkKTGjRWCJFvGWSdJeu056J+qTHIscDywLMlJQLpvnQD8QM+zSZImYKG/WfybwEeY+6W/lf8LwXPA53qcS5I0IQcNQVV9Gvh0kg9V1WcmNJMkaYIWOiIAoKo+k+SngZXzH1NV1/U0lyRpQsYKQZIvAT8M3APs6VYXYAgk6TVurBAAI+Csqqo+h5EkTd641xHcB7ypz0EkScMY94hgGfBAkn8FXty7sqre2ctUkqSJGTcEn+hzCEnScMZ919A/LPaJu4vRvgm8vtvPTVX18SRnAtcDS4G7gfdV1UuLfX5J0qtj3I+YeD7Jc93XfyXZk+S5BR72InBBVb0NOBu4JMl5wCeBT1XVKmAXsPaV/AMkSa/MWCGoqjdW1Qnd17HArwCfXeAxVVUvdIuv674KuAC4qVu/CVhzSJNLkl4Vh/Tpo1V1C3O/0A8qydFJ7gF2ArcB/wY8W1W7u022A6cdygySpFfHuBeUvWve4lHMXVew4DUFVbUHODvJicDNwFv3t9kB9rkOWAdwxhlnjDOmJOkQjPuuoV+ad3838BiwetydVNWzSe4AzgNOTLKkOypYATxxgMdsADYAjEYjL2STpJ6M+66h9y/2iZNMAf/dReA44CLmThTfDrybuXcOTQObF/vckqRXz7jvGlqR5OYkO5M8meRrSVYs8LDlwO1Jvg3cBdxWVV8HrgA+muRR4GTgmlfyD5AkvTLjvjR0LfAV4NJu+fJu3cUHekBVfRs4Zz/rtwHnLm5MSVJfxn3X0FRVXVtVu7uvLwJTPc4lSZqQcUPwVJLLu7eDHp3kcuDpPgeTJE3GuCH4AHAZ8F1gB3Mnexd9AlmSdPgZ9xzBHwPTVbULIMlS4CrmAiFJeg0b94jgx/dGAKCqnmE/J4IlSa8944bgqCQn7V3ojgjGPZqQJB3Gxv1l/ufAPye5ibmPhLgM+NPeppIkTcy4VxZfl2SGuQ+aC/Cuqnqg18kkSRMx9ss73S9+f/lL0hHmkD6GWpJ05DAEktQ4QyBJjTMEktQ4QyBJjTMEktQ4QyBJjTMEktQ4QyBJjTMEktQ4QyBJjTMEktQ4QyBJjTMEktQ4QyBJjTMEktQ4QyBJjTMEktS43kKQ5PQktyd5MMn9ST7crV+a5LYkj3S3J/U1gyRpYX0eEewGfqeq3gqcB/xWkrOA9cCWqloFbOmWJUkD6S0EVbWjqu7u7j8PPAicBqwGNnWbbQLW9DWDJGlhEzlHkGQlcA5wJ3BqVe2AuVgApxzgMeuSzCSZmZ2dncSYktSk3kOQ5PuArwEfqarnxn1cVW2oqlFVjaampvobUJIa12sIkryOuQh8uar+qlv9ZJLl3feXAzv7nEGSdHB9vmsowDXAg1X1F/O+dSsw3d2fBjb3NYMkaWFLenzu84H3Afcmuadb9zHgSuDGJGuBx4FLe5xBkrSA3kJQVf8E5ADfvrCv/UqSFscriyWpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhpnCCSpcYZAkhrXWwiSfCHJziT3zVu3NMltSR7pbk/qa/+SpPH0eUTwReCSfdatB7ZU1SpgS7csSRpQbyGoqm8Cz+yzejWwqbu/CVjT1/4lSeOZ9DmCU6tqB0B3e8qE9y9J2sdhe7I4ybokM0lmZmdnhx5Hko5Ykw7Bk0mWA3S3Ow+0YVVtqKpRVY2mpqYmNqAktWbSIbgVmO7uTwObJ7x/SdI++nz76FeBfwHenGR7krXAlcDFSR4BLu6WJUkDWtLXE1fVew/wrQv72qckafEO25PFkqTJMASS1DhDIEmNMwSS1DhDIEmNMwSS1DhDIEmNMwSS1DhDIEmNMwSS1DhDIEmNMwSS1DhDIEmNMwSS1DhDIEmNMwSS1DhDIEmNMwSS1DhDIEmNMwSS1DhDIEmNMwSS1DhDIEmNMwSS1DhDIEmNMwSS1DhDIEmNGyQESS5J8nCSR5OsH2IGSdKciYcgydHA54C3A2cB701y1qTnkCTNGeKI4Fzg0araVlUvAdcDqweYQ5LEMCE4DfjOvOXt3TpJ0gCWDLDP7GddvWyjZB2wrlt8IcnDvU7VlmXAU0MPMbRcNT30CHo5fzb3+vj+flUu2g+Os9EQIdgOnD5veQXwxL4bVdUGYMOkhmpJkpmqGg09h7QvfzaHMcRLQ3cBq5KcmeQY4D3ArQPMIUligCOCqtqd5IPA3wJHA1+oqvsnPYckac4QLw1RVd8AvjHEvgX4kpsOX/5sDiBVLztPK0lqiB8xIUmNG+SlIfUvycnAlm7xTcAeYLZb/pGqOn7etr8OjKrqgxMdUgKS7AHunbdqDbAS+N2qescgQzXGEByhqupp4GyAJJ8AXqiqq7rlFwYcTdrXf1bV2fNXJFk5zCht8qUhSWqcRwRtOi7JPfOWl+K1HBrO/J/Hf6+qXx50mgYZgjb9v0PxvecIhhtHjXvZS0OaLF8akqTGGQJJapwhkKTGeWWxJDXOIwJJapwhkKTGGQJJapwhkKTGGQJJapwhkF6BJMcn+eskDyW5P8mVQ88kLZYhkF65q6rqLcA5wPlJ3j70QNJiGAJpEZLckmRr97//dVX1vaq6HaCqXgLuBlYMO6W0OF5QJi1CkqVV9UyS44C7gJ/t/vYDSU5kLgQXVdW2IeeUFsNPH5UW57eT7P2Y5NOBVcDTSZYAXwX+0gjotcYQSGNK8nPARcBPVdX3ktwBHNt9ewPwSFVdPdB40iEzBNL4vh/Y1UXgLcB5AEn+pPvebww5nHSoPEcgjSnJ64FbgNOAh4EpYCPwJeAh4MVu089W1cZBhpQOgSGQpMb59lFJapwhkKTGGQJJapwhkKTGGQJJapwhkKTGGQJJapwhkKTG/S+vtjfSIoK3gwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(df.a2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.lowtext\n",
    "Y = df.a2\n",
    "le = LabelEncoder()\n",
    "Y = le.fit_transform(Y)\n",
    "Y = Y.reshape(-1,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train,X_test,Y_train,Y_test = train_test_split(X,Y,test_size=0.15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_words = 1000\n",
    "max_len = 150\n",
    "tok = Tokenizer(num_words=max_words)\n",
    "tok.fit_on_texts(X_train)\n",
    "sequences = tok.texts_to_sequences(X_train)\n",
    "sequences_matrix = sequence.pad_sequences(sequences,maxlen=max_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "inputs (InputLayer)          (None, 150)               0         \n",
      "_________________________________________________________________\n",
      "embedding_1 (Embedding)      (None, 150, 50)           50000     \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 64)                29440     \n",
      "_________________________________________________________________\n",
      "FC1 (Dense)                  (None, 256)               16640     \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "out_layer (Dense)            (None, 1)                 257       \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 1)                 0         \n",
      "=================================================================\n",
      "Total params: 96,337\n",
      "Trainable params: 96,337\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "def RNN():\n",
    "    inputs = Input(name='inputs',shape=[max_len])\n",
    "    layer = Embedding(max_words,50,input_length=max_len)(inputs)\n",
    "    layer = LSTM(64)(layer)\n",
    "    layer = Dense(256,name='FC1')(layer)\n",
    "    layer = Activation('relu')(layer)\n",
    "    layer = Dropout(0.5)(layer)\n",
    "    layer = Dense(1,name='out_layer')(layer)\n",
    "    layer = Activation('sigmoid')(layer)\n",
    "    model = Model(inputs=inputs,outputs=layer)\n",
    "    return model\n",
    "\n",
    "model = RNN()\n",
    "model.summary()\n",
    "model.compile(loss='binary_crossentropy',optimizer=RMSprop(),metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 77 samples, validate on 9 samples\n",
      "Epoch 1/20\n",
      "77/77 [==============================] - 1s 16ms/step - loss: 0.6905 - acc: 0.5325 - val_loss: 0.6694 - val_acc: 0.6667\n",
      "Epoch 2/20\n",
      "77/77 [==============================] - 0s 5ms/step - loss: 0.6710 - acc: 0.6104 - val_loss: 0.6139 - val_acc: 0.8889\n",
      "Epoch 3/20\n",
      "77/77 [==============================] - 0s 5ms/step - loss: 0.5926 - acc: 0.7532 - val_loss: 0.5194 - val_acc: 0.8889\n",
      "Epoch 4/20\n",
      "77/77 [==============================] - 0s 5ms/step - loss: 0.5068 - acc: 0.8831 - val_loss: 0.3417 - val_acc: 1.0000\n",
      "Epoch 5/20\n",
      "77/77 [==============================] - 0s 5ms/step - loss: 0.5447 - acc: 0.8831 - val_loss: 0.3316 - val_acc: 0.8889\n",
      "Epoch 6/20\n",
      "77/77 [==============================] - 0s 6ms/step - loss: 0.2810 - acc: 0.9091 - val_loss: 0.1651 - val_acc: 1.0000\n",
      "Epoch 7/20\n",
      "77/77 [==============================] - 0s 5ms/step - loss: 0.3967 - acc: 0.8961 - val_loss: 0.2276 - val_acc: 1.0000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f82e425dd30>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(sequences_matrix,Y_train,batch_size=16,epochs=20,\n",
    "          validation_split=0.1,callbacks=[EarlyStopping(monitor='val_loss',min_delta=0.0001)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "16/16 [==============================] - 0s 1ms/step\n"
     ]
    }
   ],
   "source": [
    "test_sequences = tok.texts_to_sequences(X_test)\n",
    "test_sequences_matrix = sequence.pad_sequences(test_sequences,maxlen=max_len)\n",
    "accr = model.evaluate(test_sequences_matrix,Y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test set\n",
      "  Loss: 0.381\n",
      "  Accuracy: 0.875\n"
     ]
    }
   ],
   "source": [
    "print('Test set\\n  Loss: {:0.3f}\\n  Accuracy: {:0.3f}'.format(accr[0],accr[1]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}