tensorflow/Text Classification with MLP (MultiLayer Perceptron) in Tensorflow Keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Classification with MLP (MultiLayer Perceptron) in Tensorflow Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MLP is defined as a Neural Network with at least one dense layer added in between. MLP is the most basic concept in Feed Forward Neural Network. Feed Forward Neural Network is simply the network where the direction of layer is one-way from input to output layers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MLP is useful in calssiciation model, but not so much in NLP model, as the length of the input is fixed (cannot be changed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we will use the texts_to_matrx() function in Keras to perform text-classification.  texts_to_matrx() function has 4 different modes: 'binary', 'count', 'freq', 'tfidf'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification of twenty e-mail newsgroup using Keras Text_to_matrix function "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Dropout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sklearn offers 18,846 e-mails about 20 different news group. So we are going to use that for a learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "newsdata = fetch_20newsgroups(subset = 'train') # 'subset = train' returns only train dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['data', 'filenames', 'target_names', 'target', 'DESCR'])\n"
     ]
    }
   ],
   "source": [
    "print(newsdata.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of e-mails in training set : 11314\n"
     ]
    }
   ],
   "source": [
    "print('Number of e-mails in training set : {}'.format(len(newsdata.data)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of target classes : 20\n",
      "['alt.atheism', 'comp.graphics', 'comp.os.ms-windows.misc', 'comp.sys.ibm.pc.hardware', 'comp.sys.mac.hardware', 'comp.windows.x', 'misc.forsale', 'rec.autos', 'rec.motorcycles', 'rec.sport.baseball', 'rec.sport.hockey', 'sci.crypt', 'sci.electronics', 'sci.med', 'sci.space', 'soc.religion.christian', 'talk.politics.guns', 'talk.politics.mideast', 'talk.politics.misc', 'talk.religion.misc']\n"
     ]
    }
   ],
   "source": [
    "print('Number of target classes : {}'.format(len(newsdata.target_names)))\n",
    "print(newsdata.target_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have 20 different classes in this classification task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>email</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>From: lerxst@wam.umd.edu (where's my thing)\\nS...</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>From: guykuo@carson.u.washington.edu (Guy Kuo)...</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>From: twillis@ec.ecn.purdue.edu (Thomas E Will...</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>From: jgreen@amber (Joe Green)\\nSubject: Re: W...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>From: jcm@head-cfa.harvard.edu (Jonathan McDow...</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               email  target\n",
       "0  From: lerxst@wam.umd.edu (where's my thing)\\nS...       7\n",
       "1  From: guykuo@carson.u.washington.edu (Guy Kuo)...       4\n",
       "2  From: twillis@ec.ecn.purdue.edu (Thomas E Will...       4\n",
       "3  From: jgreen@amber (Joe Green)\\nSubject: Re: W...       1\n",
       "4  From: jcm@head-cfa.harvard.edu (Jonathan McDow...      14"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert data into Pandas Dataframe for easy handling\n",
    "data = pd.DataFrame(newsdata.data, columns = ['email']) \n",
    "data['target'] = pd.Series(newsdata.target) # add target column\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD/CAYAAAD/qh1PAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAFHtJREFUeJzt3XuwXWV5x/HvAxG8QEmAw8UkEJUo6tQgPYN4qRfihUs1qUKrdjSltPmjqHjpaLzMUB1rYy+izlTajBGDRQFRhlQQxQA6tiVyuBjAYBMjJKcBclSIF6oWfPrHejPuHE6y1zlnn3N23n4/M3v2Wu9617vfvc/Ob7/r3WuvRGYiSarXfjPdAUnS1DLoJalyBr0kVc6gl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZUz6CWpcrNmugMAhx9+eC5YsGCmuyFJ+5RbbrnlR5k50K1eXwT9ggULGBoamuluSNI+JSLubVPPqRtJqpxBL0mVM+glqXIGvSRVrlXQR8TsiLgiIu6OiI0R8fyIODQirouITeV+TqkbEfHJiNgcERsi4sSpfQqSpL1pO6L/BHBtZh4PLAI2AiuAdZm5EFhX1gFOAxaW23Lgwp72WJI0Ll2DPiJ+B3gxsBogM3+dmQ8BS4A1pdoaYGlZXgJcnI2bgNkRcXTPey5JaqXNiP6pwAhwUUTcFhGfjognAUdm5n0A5f6IUn8usK1j/+FSJkmaAW1+MDULOBF4a2auj4hP8NtpmrHEGGWP+R/II2I5zdQOxxxzzG7bFqy4umun7ll5xl63d2uj2/6SVIs2QT8MDGfm+rJ+BU3QPxARR2fmfWVqZkdH/fkd+88Dto9uNDNXAasABgcHH/NB0A8m+2HRiw8sSZqsrkGfmfdHxLaIeEZmfh9YDHyv3JYBK8v9VWWXtcBbIuJS4HnAzl1TPBq/fjm66YcPPT84pYlpe62btwKXRMQBwBbgbJr5/csj4hxgK3BWqXsNcDqwGXi41JX6Qi0fetJ4tAr6zLwdGBxj0+Ix6iZw7iT7JUnqkb64eqWk8emXKT3tG7wEgiRVzqCXpMo5dSNpwpz+2Tc4opekyhn0klQ5g16SKuccvaQZ44/HpocjekmqnEEvSZUz6CWpcga9JFXOoJekyhn0klQ5T6+UtE/zMgzdOaKXpMoZ9JJUOYNekipn0EtS5Qx6SaqcQS9JlTPoJalyBr0kVc6gl6TKGfSSVDmDXpIq1yroI+KeiLgjIm6PiKFSdmhEXBcRm8r9nFIeEfHJiNgcERsi4sSpfAKSpL0bz4j+ZZl5QmYOlvUVwLrMXAisK+sApwELy205cGGvOitJGr/JTN0sAdaU5TXA0o7yi7NxEzA7Io6exONIkiahbdAn8PWIuCUilpeyIzPzPoByf0Qpnwts69h3uJTtJiKWR8RQRAyNjIxMrPeSpK7aXo/+hZm5PSKOAK6LiLv3UjfGKMvHFGSuAlYBDA4OPma7JKk3Wo3oM3N7ud8BXAmcBDywa0qm3O8o1YeB+R27zwO296rDkqTx6Rr0EfGkiDh41zLwSuBOYC2wrFRbBlxVltcCby5n35wM7Nw1xSNJmn5tpm6OBK6MiF31P5+Z10bEzcDlEXEOsBU4q9S/Bjgd2Aw8DJzd815LklrrGvSZuQVYNEb5j4HFY5QncG5PeidJmjR/GStJlWt71o0kVWvBiqv3uv2elWdMU0+mhiN6SaqcQS9JlTPoJalyBr0kVc6gl6TKedaNJPVAP5+544hekipn0EtS5Qx6SaqcQS9JlTPoJalyBr0kVc6gl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZUz6CWpcga9JFXOoJekyhn0klQ5g16SKtc66CNi/4i4LSK+UtafEhHrI2JTRFwWEQeU8gPL+uayfcHUdF2S1MZ4RvTnARs71j8KXJCZC4EHgXNK+TnAg5l5HHBBqSdJmiGtgj4i5gFnAJ8u6wGcAlxRqqwBlpblJWWdsn1xqS9JmgFtR/QfB94N/KasHwY8lJmPlPVhYG5ZngtsAyjbd5b6kqQZ0DXoI+IPgB2ZeUtn8RhVs8W2znaXR8RQRAyNjIy06qwkafzajOhfCLwmIu4BLqWZsvk4MDsiZpU684DtZXkYmA9Qth8C/GR0o5m5KjMHM3NwYGBgUk9CkrRnXYM+M9+bmfMycwHweuD6zPwT4AbgzFJtGXBVWV5b1inbr8/Mx4zoJUnTYzLn0b8HeGdEbKaZg19dylcDh5XydwIrJtdFSdJkzOpe5bcy80bgxrK8BThpjDq/BM7qQd8kST3gL2MlqXIGvSRVzqCXpMqNa45ekjQ1Fqy4umude1aeMaG2HdFLUuUMekmqnEEvSZUz6CWpcga9JFXOoJekyhn0klQ5g16SKmfQS1LlDHpJqpxBL0mVM+glqXIGvSRVzqCXpMoZ9JJUOYNekipn0EtS5Qx6SaqcQS9JlTPoJalyBr0kVc6gl6TKdQ36iHh8RHwnIr4bEXdFxAdL+VMiYn1EbIqIyyLigFJ+YFnfXLYvmNqnIEnamzYj+l8Bp2TmIuAE4NSIOBn4KHBBZi4EHgTOKfXPAR7MzOOAC0o9SdIM6Rr02fh5WX1cuSVwCnBFKV8DLC3LS8o6ZfviiIie9ViSNC6t5ugjYv+IuB3YAVwH/AB4KDMfKVWGgblleS6wDaBs3wkc1stOS5LaaxX0mfloZp4AzANOAp45VrVyP9boPUcXRMTyiBiKiKGRkZG2/ZUkjdO4zrrJzIeAG4GTgdkRMatsmgdsL8vDwHyAsv0Q4CdjtLUqMwczc3BgYGBivZckddXmrJuBiJhdlp8AvBzYCNwAnFmqLQOuKstryzpl+/WZ+ZgRvSRpeszqXoWjgTURsT/NB8PlmfmViPgecGlEfBi4DVhd6q8GPhcRm2lG8q+fgn5LklrqGvSZuQF47hjlW2jm60eX/xI4qye9kyRNmr+MlaTKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZUz6CWpcga9JFXOoJekyhn0klQ5g16SKmfQS1LlDHpJqpxBL0mVM+glqXIGvSRVzqCXpMoZ9JJUOYNekipn0EtS5Qx6SaqcQS9JlTPoJalyBr0kVc6gl6TKdQ36iJgfETdExMaIuCsizivlh0bEdRGxqdzPKeUREZ+MiM0RsSEiTpzqJyFJ2rM2I/pHgHdl5jOBk4FzI+JZwApgXWYuBNaVdYDTgIXlthy4sOe9liS11jXoM/O+zLy1LP8M2AjMBZYAa0q1NcDSsrwEuDgbNwGzI+LonvdcktTKuOboI2IB8FxgPXBkZt4HzYcBcESpNhfY1rHbcCmTJM2A1kEfEQcBXwLenpk/3VvVMcpyjPaWR8RQRAyNjIy07YYkaZxaBX1EPI4m5C/JzC+X4gd2TcmU+x2lfBiY37H7PGD76DYzc1VmDmbm4MDAwET7L0nqos1ZNwGsBjZm5sc6Nq0FlpXlZcBVHeVvLmffnAzs3DXFI0mafrNa1Hkh8Cbgjoi4vZS9D1gJXB4R5wBbgbPKtmuA04HNwMPA2T3tsSRpXLoGfWZ+m7Hn3QEWj1E/gXMn2S9JUo/4y1hJqpxBL0mVM+glqXIGvSRVzqCXpMoZ9JJUOYNekipn0EtS5Qx6SaqcQS9JlTPoJalyBr0kVc6gl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZUz6CWpcga9JFXOoJekyhn0klQ5g16SKmfQS1LlDHpJqpxBL0mV6xr0EfGZiNgREXd2lB0aEddFxKZyP6eUR0R8MiI2R8SGiDhxKjsvSequzYj+s8Cpo8pWAOsycyGwrqwDnAYsLLflwIW96aYkaaK6Bn1mfgv4yajiJcCasrwGWNpRfnE2bgJmR8TRveqsJGn8JjpHf2Rm3gdQ7o8o5XOBbR31hkuZJGmG9PrL2BijLMesGLE8IoYiYmhkZKTH3ZAk7TLRoH9g15RMud9RyoeB+R315gHbx2ogM1dl5mBmDg4MDEywG5KkbiYa9GuBZWV5GXBVR/mby9k3JwM7d03xSJJmxqxuFSLiC8BLgcMjYhg4H1gJXB4R5wBbgbNK9WuA04HNwMPA2VPQZ0nSOHQN+sx8wx42LR6jbgLnTrZTkqTe8ZexklQ5g16SKmfQS1LlDHpJqpxBL0mVM+glqXIGvSRVzqCXpMoZ9JJUOYNekipn0EtS5Qx6SaqcQS9JlTPoJalyBr0kVc6gl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZUz6CWpcga9JFXOoJekyhn0klS5KQn6iDg1Ir4fEZsjYsVUPIYkqZ2eB31E7A/8E3Aa8CzgDRHxrF4/jiSpnakY0Z8EbM7MLZn5a+BSYMkUPI4kqYWpCPq5wLaO9eFSJkmaAZGZvW0w4izgVZn552X9TcBJmfnWUfWWA8vL6jOA7++l2cOBH02ya7W00Q996Jc2+qEP/dJGP/ShX9rohz5MVxvHZuZA11Yys6c34PnA1zrW3wu8d5JtDvWgX1W00Q996Jc2+qEP/dJGP/ShX9rohz70UxuZOSVTNzcDCyPiKRFxAPB6YO0UPI4kqYVZvW4wMx+JiLcAXwP2Bz6TmXf1+nEkSe30POgBMvMa4JoeNrnKNvqqD/3SRj/0oV/a6Ic+9Esb/dCHfmqj91/GSpL6i5dAkKTKGfSSVLkpmaOvQccZQ9sz8xsR8UbgBcBGYFVm/u+MdlD7rIg4CcjMvLlcHuRU4O7y3dY+KyIuzsw3z3Q/JiIijqf5Bf9cIIHtwNrM3DijHesR5+j3ICIuofkgfCLwEHAQ8GVgMc3rtqxFG08D/hCYDzwCbAK+kJk7p6rfUyEi3gZcmZnbulaeJhHxIprLbdyZmV+f5sc+niYQ1mfmzzvKT83Ma7vsez7NdaBmAdcBzwNuBF5O8/uTv2nx+M8DNmbmTyPiCcAK4ETge8BHpuP9FRGjT5kO4GXA9QCZ+Zqp7kOvRMR7gDfQXK5luBTPoxnoXZqZK2eqb71i0O9BRGzIzOdExCzgv4EnZ+ajERHAdzPzOV32fxvwauCbwOnA7cCDNMH/l5l545Q+gR6KiJ3AL4AfAF8AvpiZIz1o94jM3NGy7ncy86Sy/BfAucCVwCuBf5uuf4zl73ouzZHdCcB5mXlV2XZrZp7YZf87yn4HAvcD8zoCe32391Vp4y5gUTmVeRXwMHAFzSBkUWa+duLPECLi7My8qEudW2k+WD5NMwIOmvfG6wEy85uT6cNkRMRhmfnjcdT/L+DZo4/Sy1H9XZm5sNd9nHa9+NVVr2/AIcBK4G7gx+W2sZTNbrH/qaPaWg1sAD4PHNmyD3cCBwBzgJ8Bh5byx9OMprrtfwewf1l+InBjWT4GuK1Hr9NXW9Q5CriQ5oqihwF/Xfp2OXB0y8e5jeb7nFeW13IEuBZYBhzcso1DR90OA+4pr++hbfrQsXwzMFCWnwTc0bIPvwP8LfA54I2jtn2qZRt3AAeV5QXAEE3Y79bHls/jtlHbbm/Zh40dy7dOpI0u7W9tUWc/4B00RyUnlLItE3isW4EPAE+bYF9XAoeX5UFgC7AZuBd4Scs27qa5lMDo8mOB77dsYxC4AfhXmiP464Cd5b363JZtHAR8CLir7DsC3AT86WT/pv06R385zSHgSzPzfoCIOIomWL4IvKLL/h+hCSKAfwTuoxldvxb4F2Bpiz6spnkD7A+8H/hiRGwBTqY5xGtjFvAozejtYIDM3BoRj2u5PxGxpxFi0IwMu/kscDVNIN4AXAKcQTMf+c+0u7JoZuZvgK8DXy/9P43mcPcfgO7X2miu13HvqLK5NP/QE3hql/33i4g5NAETWY4oMvMXEfFIi8cHuIhm+uxLwJ9FxOtoAv9XNH/XNvbPMl2TmfdExEuBKyLiWJq/STe/jognZubDwO/tKoyIQ4DftOzDnR2j7u9GxGBmDkXE04FW3x1FxIY9bQKO7LZ/eT9cEBFfLPcPMLHv/OYAs4EbIuJ+mqOCyzJze8v9z8jMXf/nxd8Df5zNdx9PpxnYDbZo4+3AuojYxG8vyHgMcBzwlpb9+BRwPs1z+Q/gHZn5iohYXLY9v0Ubl9Acpb4K+COaf7OXAh+IiKdn5vta9uWxJvtJMRU39vIpurdtHXVu7Vi+fdS21iMe4Mk0UzaUP+CZNBdoa7PveTRHEatoPjDOLuUDwLfG0YdHaT70bhjj9j8t9u8cQW4dta3tCHKPI1XgCS3b+CuaD9/f7Sj74Theh3toRms/LPdHlfKDxvE8Rr8X3g/8O83Rxa0t27ieMoLtKJsFXAw82mL/A/dQfnjna9OljUNoPsB/AKynCfctNNOEi1q28QDNQOHYUbcFNCcgtPq7dLR3Bs33A+Pdr/Pf6u/ThOL95f29vMX+dwOzyvJNo7a1OtIrdfej+bB/Xfl3fjLliLzl/nv7d9bqCJ5mSrhz/eaOvt093td2t7Yms/NU3WhGju+mY5qFZpTxHuAbLfYfBt4JvKv8A4iObRum8Xk8u7xpjp9EG3cCC/ewbdt43jzAh0dtazvl8fQevR7zaI7IPkZzhDPuQ/0x2nwi8JSWdTcC+40qW0ZzqHzvOJ7DUXvY9sLpem+VxzsYWERzZNBqSrJj39XAi/aw7fPT+Bwe8wFLcxR9KnBRi/3fWvLiFJppyY8DLwY+CHxuGp/Hf9JMbZ5Fc+S6tJS/hJYXJqM5EnhRWX41u18cstUU0h7bnq4XYpwv2hzgo+XT+ifltrGUzWmx//mjbrvmc48CLp7p5zfO1+JM4Bl72La0xf4foswpjyo/Drhihp7Tq2nmHu+f5sf9O+DlY5SfCmya6b/1/8cbzVktk23jpcBlNN8l3UFz+ZXllJH+ND2PRTTX9/oqcDzwCZqz9e4CXtCyjecA3yn7fZsywKKZBXjbZPq3z5110+aMgKncv5/sy69FOcvkaZl5Zz/8TfqhD9rdvvz+7nU/Jv1a7INBvzUzj5mp/ftJLa9FP/SjH/qg3fn+7l0bfXnWzWTPCJjs/v2klteiH/rRD33Q7nx/97aNPenLoKd5Uq+i+YFRp6D5wmKq9+8ntbwW/dCPfuiDduf7u7dtjKlfg/4rNF8g3j56Q0TcOA3795NaXot+6Ec/9EG78/3d2zbGtM/N0UuSxsfLFEtS5Qx6SaqcQS9JlTPoJalyBr0kVe7/AGw3z/4mZsNGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Simple visualization on target classes\n",
    "data['target'].value_counts().plot(kind='bar');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Splitting data into predictor/response for slotting into model, and also calling testset as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_email = data['email'] # predictor in training set\n",
    "train_label = data['target'] # response in training set\n",
    "\n",
    "newsdata_test = fetch_20newsgroups(subset='test', shuffle=True) \n",
    "test_email = newsdata_test.data # predictor in test set\n",
    "test_label = newsdata_test.target # response in training set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set neccesary varialbes \n",
    "> max_words: number of maximum words that the models gonna get feed\n",
    "> num_classes: number of labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_words = 10000 # Max number of word that we will input in training\n",
    "num_classes = 20 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data preprocessing function\n",
    "def prepare_data(train_data, test_data, mode):\n",
    "    t = Tokenizer(num_words = max_words) # Use the number of words in 'max_words' \n",
    "    t.fit_on_texts(train_data)\n",
    "    X_train = t.texts_to_matrix(train_data, mode=mode) # matrix that has the size of \"number of samples × max_words\"\n",
    "    X_test = t.texts_to_matrix(test_data, mode=mode) # matrix that has the size of \"number of samples × max_words\"\n",
    "    return X_train, X_test, t.index_word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, index_to_word = prepare_data(train_email, test_email, 'binary') # For now, let's use binary mode\n",
    "y_train = to_categorical(train_label, num_classes) # One-hot encoding\n",
    "y_test = to_categorical(test_label, num_classes) # One-hot encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(11314, 10000)\n",
      "(7532, 10000)\n",
      "(11314, 20)\n",
      "(7532, 20)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The 1st Frequent word : the\n",
      "9999th Frequent word : mic\n"
     ]
    }
   ],
   "source": [
    "print('The 1st Frequent word : {}'.format(index_to_word[1]))\n",
    "print('9999th Frequent word : {}'.format(index_to_word[9999]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Use Multilayer Perceptron "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![imge]('./img/multilayerperceptron.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The image above shows the structure of the code below. There are 4 layers in total: Input layer that has the size of the max_words, first hidden layer with 256 neurons, second hidden layer with 128 neurons, and the output layer with the number of classes we want to predict. This network is called Deep Neural Networ (DNN) as it has more than one hidden layer "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_and_evaluate(X_train, y_train, X_test, y_test):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(256, input_shape=(max_words,), activation='relu'))\n",
    "    model.add(Dropout(0.5)) # To prevent overfitting\n",
    "    model.add(Dense(128, activation='relu'))\n",
    "    model.add(Dropout(0.5)) # To prevent overfitting\n",
    "    model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "    model.fit(X_train, y_train, batch_size=128, epochs=5, verbose=1, validation_split=0.1)\n",
    "    score = model.evaluate(X_test, y_test, batch_size=128, verbose=0)\n",
    "    return score[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Train the Model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get the result for all 4 available mode for text_to_matrix() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "80/80 [==============================] - 2s 21ms/step - loss: 2.2875 - accuracy: 0.3400 - val_loss: 0.9542 - val_accuracy: 0.8357\n",
      "Epoch 2/5\n",
      "80/80 [==============================] - 1s 18ms/step - loss: 0.8686 - accuracy: 0.7608 - val_loss: 0.4607 - val_accuracy: 0.8869\n",
      "Epoch 3/5\n",
      "80/80 [==============================] - 1s 18ms/step - loss: 0.4236 - accuracy: 0.8885 - val_loss: 0.3578 - val_accuracy: 0.9055\n",
      "Epoch 4/5\n",
      "80/80 [==============================] - 1s 18ms/step - loss: 0.2577 - accuracy: 0.9352 - val_loss: 0.3181 - val_accuracy: 0.9117\n",
      "Epoch 5/5\n",
      "80/80 [==============================] - 2s 21ms/step - loss: 0.1665 - accuracy: 0.9608 - val_loss: 0.2896 - val_accuracy: 0.9178\n",
      "binary Test Accuacy: 0.8284652233123779\n",
      "Epoch 1/5\n",
      "80/80 [==============================] - 3s 43ms/step - loss: 2.7406 - accuracy: 0.2414 - val_loss: 1.5910 - val_accuracy: 0.7509\n",
      "Epoch 2/5\n",
      "80/80 [==============================] - 3s 34ms/step - loss: 1.4450 - accuracy: 0.6223 - val_loss: 0.7450 - val_accuracy: 0.8436\n",
      "Epoch 3/5\n",
      "80/80 [==============================] - 2s 31ms/step - loss: 0.7898 - accuracy: 0.8008 - val_loss: 0.4838 - val_accuracy: 0.8816\n",
      "Epoch 4/5\n",
      "80/80 [==============================] - 3s 32ms/step - loss: 0.5293 - accuracy: 0.8676 - val_loss: 0.4325 - val_accuracy: 0.8887\n",
      "Epoch 5/5\n",
      "80/80 [==============================] - 2s 28ms/step - loss: 0.4255 - accuracy: 0.9055 - val_loss: 0.4135 - val_accuracy: 0.8896\n",
      "count Test Accuacy: 0.8130642771720886\n",
      "Epoch 1/5\n",
      "80/80 [==============================] - 4s 51ms/step - loss: 2.2337 - accuracy: 0.3604 - val_loss: 0.7676 - val_accuracy: 0.8489\n",
      "Epoch 2/5\n",
      "80/80 [==============================] - 4s 44ms/step - loss: 0.7850 - accuracy: 0.7812 - val_loss: 0.4647 - val_accuracy: 0.8896\n",
      "Epoch 3/5\n",
      "80/80 [==============================] - 3s 37ms/step - loss: 0.4448 - accuracy: 0.8814 - val_loss: 0.3527 - val_accuracy: 0.9072\n",
      "Epoch 4/5\n",
      "80/80 [==============================] - 3s 33ms/step - loss: 0.2847 - accuracy: 0.9290 - val_loss: 0.3338 - val_accuracy: 0.9143\n",
      "Epoch 5/5\n",
      "80/80 [==============================] - 3s 31ms/step - loss: 0.2074 - accuracy: 0.9490 - val_loss: 0.3268 - val_accuracy: 0.9143\n",
      "tfidf Test Accuacy: 0.8278014063835144\n",
      "Epoch 1/5\n",
      "80/80 [==============================] - 2s 31ms/step - loss: 2.9781 - accuracy: 0.0935 - val_loss: 2.9267 - val_accuracy: 0.2120\n",
      "Epoch 2/5\n",
      "80/80 [==============================] - 2s 27ms/step - loss: 2.7249 - accuracy: 0.2078 - val_loss: 2.4103 - val_accuracy: 0.4435\n",
      "Epoch 3/5\n",
      "80/80 [==============================] - 2s 24ms/step - loss: 2.2209 - accuracy: 0.3172 - val_loss: 1.9297 - val_accuracy: 0.5433\n",
      "Epoch 4/5\n",
      "80/80 [==============================] - 2s 25ms/step - loss: 1.7842 - accuracy: 0.4483 - val_loss: 1.5156 - val_accuracy: 0.6343\n",
      "Epoch 5/5\n",
      "80/80 [==============================] - 2s 23ms/step - loss: 1.4179 - accuracy: 0.5653 - val_loss: 1.2084 - val_accuracy: 0.7253\n",
      "freq Test Accuacy: 0.674986720085144\n"
     ]
    }
   ],
   "source": [
    "modes = ['binary', 'count', 'tfidf', 'freq']\n",
    "\n",
    "for mode in modes: \n",
    "    X_train, X_test, _ = prepare_data(train_email, test_email, mode) # PreProcess data per each modes\n",
    "    score = fit_and_evaluate(X_train, y_train, X_test, y_test) \n",
    "    print(mode+' Test Accuacy:', score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Around 81~82% of accuracy accorss all modes, except 67% at 'freq' mode, which tells it's not a good algorithm to solve this problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}