bow_classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NN\n",
    "\n",
    "* We will cover the essentials of NN during the lecture\n",
    "* Those absent, try eg. [this tutorial](https://www.cs.toronto.edu/~jlucas/teaching/csc411/lectures/tut5_handout.pdf)\n",
    "* For everyone interested, [here](https://gombru.github.io/2018/05/23/cross_entropy_loss/) is some reading about the loss functions\n",
    "* And [here](https://github.com/Jaewan-Yun/optimizer-visualization) is the visualization of different optimizers I showed\n",
    "* We also walked through several optimization techniques, if you missed the lecture on that, I suggest you watch these: [Momentum](https://www.youtube.com/watch?v=N18Km9YIIug) [RMSProp](https://www.youtube.com/watch?v=XhZahXzEuNo) and [Adam](https://www.youtube.com/watch?v=JXQT_vxqwIs). These videos are by Hinton and Ng, who are trustable sources. The slides are [here](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf). Further reading (from a trustable source) is also to be found [here](https://ruder.io/optimizing-gradient-descent/index.html#adagrad). The [Adam paper](https://arxiv.org/pdf/1412.6980.pdf) is for those especially interested in the topic.\n",
    "\n",
    "# Bag-of-words document classification\n",
    "\n",
    "* BoW is the simplest way to do classification: Feature vector goes in, decision falls out.\n",
    "\n",
    "* Feature vector: a vector with as many dimensions as we have unique features, and a non-zero value set for every feature present in our example\n",
    "* Binary features: 1/0\n",
    "\n",
    "In the following we work with the IMDB data, have a look on [how to read it in](read_imdb.ipynb). Here we just read the ready data in.\n",
    "\n",
    "# IMDB data\n",
    "\n",
    "* Movie review sentiment positive/negative\n",
    "* Some 25,000 examples, 50:50 split\n",
    "* Current state-of-the-art is about 95% accuracy\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-03-03 09:23:15--  https://raw.githubusercontent.com/TurkuNLP/Deep_Learning_in_LangTech_course/master/data/imdb_train.json\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.84.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.84.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 33944099 (32M) [text/plain]\n",
      "Saving to: ‘imdb_train.json.3’\n",
      "\n",
      "imdb_train.json.3   100%[===================>]  32,37M  15,0MB/s    in 2,2s    \n",
      "\n",
      "2020-03-03 09:23:19 (15,0 MB/s) - ‘imdb_train.json.3’ saved [33944099/33944099]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://raw.githubusercontent.com/TurkuNLP/Deep_Learning_in_LangTech_course/master/data/imdb_train.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'class': 'neg', 'text': 'About 15 minutes in, my wife was already wanting to leave. Not so much because of the material, but the lack thereof. They decided to fill in the blanks where the funny stuff should\\'ve been with as much language and absolutely vulgar talk as they could. When this would let up (very rare), we\\'d sit back and watch (not laughing, mind you) and wait for the next gross-out or offensive remark(s). After about 35 minutes, we both got up and left. Everything we\\'d read said how great this was. The trailer looked good and Roger Ebert actually called it \\\\intelligent\\\\\" and said it wasn\\'t a crude sex comedy. Did he go to the right movie? Along with Be Cool, it\\'s the only other movie I\\'ve ever walked out on...and I have no regrets. I\\'m sick of trying to go see comedies in America.\"'}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import random\n",
    "with open(\"imdb_train.json\") as f:\n",
    "    data=json.load(f)\n",
    "random.shuffle(data) #play it safe!\n",
    "print(data[0]) #Every item is a dictionary with `text` and `class` keys, here's the first one:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To learn on this data, we will need a few steps:\n",
    "\n",
    "* Build a data matrix with dimensionality (number of examples, number of possible features), and a value for each feature, 0/1 for binary features\n",
    "* Build a class label matrix (number of examples, number of classes) with the correct labels for the examples, setting 1 for the correct class, and 0 for others\n",
    "\n",
    "It is quite useless to do all this ourselves, so we will use ready-made classes and functions mostly from scikit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['About 15 minutes in, my wife was already wanting to leave. Not so much because of the material, but the lack thereof. They decided to fill in the blanks where the funny stuff should\\'ve been with as much language and absolutely vulgar talk as they could. When this would let up (very rare), we\\'d sit back and watch (not laughing, mind you) and wait for the next gross-out or offensive remark(s). After about 35 minutes, we both got up and left. Everything we\\'d read said how great this was. The trailer looked good and Roger Ebert actually called it \\\\intelligent\\\\\" and said it wasn\\'t a crude sex comedy. Did he go to the right movie? Along with Be Cool, it\\'s the only other movie I\\'ve ever walked out on...and I have no regrets. I\\'m sick of trying to go see comedies in America.\"', 'i chose to see the this film on the day it opened nationally in france, as a personal way for myself to reflect on what had happened a year previous; the collection works as intended: it provokes a whirlwind of thoughts and emotions, working as an intellectual hommage, never stooping to cheap sentimentality nor knee-jerk reactionism.  there have been many allegations made that the film is anti-american: while i cannot speak for everyone in this regard, i am one american who found such statements to be completely untrue. people make much noise about the egyptian segment, by Chahine, because it voices perspectives of palestinian suicide bombers asserting that civilians in a democracy are \\\\fair targets\\\\\" for they elect the governments the bombers are seeking to attack, but this ignores much else in the piece: several perspectives are discussed, no one being held up as the truth, and critics--if they even saw the piece--seem to forget the fondness and warm dialogue that takes place between the director and the ghost of the american dialogue, and the director\\'s intense sadness upon hearing of the tragedy.  pretty much all of the films are beautiful, thoughtful & inspiring, in particular the brilliant work by Mahkmalbaf, Tanovic, Loach & Inarritu. Nair, good as usual, effectively tells a true story of an injustice committed against a muslim family in the wake of anti-islam hysteria that swept--and still sweeps--the states. i did find Gitai\\'s piece a bit vulgarly loud and simple in it\\'s critic of media hysteria in the face of terrorism, and Penn\\'s piece was too impressionistic and elliptical for my tastes, though i had expected to like it. Borgnine is very good and brave in it. SPOILER WARNING: one reviewer below incorrectly read the falling of the towers as being a happy moment for the character; my read is rather that the falling of the towers is what, because light floods his room, keys him into the loss in his life that he refused to recognize. again this is a sort of impressionistic piece, for we know that if the towers were really blocking the light to this man\\'s flat, then there would have been nothing but smoke and ash, not light, flooding through his window.\"']\n",
      "['neg', 'pos']\n"
     ]
    }
   ],
   "source": [
    "# We need to gather the texts, into a list\n",
    "texts=[one_example[\"text\"] for one_example in data]\n",
    "labels=[one_example[\"class\"] for one_example in data]\n",
    "print(texts[:2])\n",
    "print(labels[:2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape= (25000, 74849)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "vectorizer=CountVectorizer(max_features=100000,binary=True,ngram_range=(1,1))\n",
    "feature_matrix=vectorizer.fit_transform(texts)\n",
    "print(\"shape=\",feature_matrix.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have the feature matrix done! Next thing we need is the class labels to be predicted in one-hot encoding. This means:\n",
    "\n",
    "* one row for every example\n",
    "* one column for every possible class label\n",
    "* exactly one column has 1 for every example, corresponding to the desired class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "class_numbers shape= (25000,)\n",
      "class labels ['neg' 'pos']\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "label_encoder=LabelEncoder() #Turns class labels into integers\n",
    "class_numbers=label_encoder.fit_transform(labels)\n",
    "\n",
    "print(\"class_numbers shape=\",class_numbers.shape)\n",
    "print(\"class labels\",label_encoder.classes_) #this will let us translate back from indices to labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The data is ready, we need to build the network now\n",
    "* Input\n",
    "* Hidden Dense layer with some kind of non-linearity, and a suitable number of nodes\n",
    "* Output Dense layer with the softmax activation (normalizes output to distribution) and as many nodes as there are classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "/home/ginter/venv-jlab/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/home/ginter/venv-jlab/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/home/ginter/venv-jlab/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/home/ginter/venv-jlab/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/home/ginter/venv-jlab/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/home/ginter/venv-jlab/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "/home/ginter/venv-jlab/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/home/ginter/venv-jlab/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/home/ginter/venv-jlab/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/home/ginter/venv-jlab/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/home/ginter/venv-jlab/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/home/ginter/venv-jlab/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "from keras.models import Model\n",
    "from keras.layers import Input, Dense\n",
    "\n",
    "example_count,feature_count=feature_matrix.shape\n",
    "class_count=len(label_encoder.classes_)\n",
    "\n",
    "inp=Input(shape=(feature_count,))\n",
    "hidden=Dense(200,activation=\"tanh\")(inp)\n",
    "outp=Dense(class_count,activation=\"softmax\")(hidden)\n",
    "model=Model(inputs=[inp], outputs=[outp])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...it's **this** simple...!\n",
    "\n",
    "Once the model is constructed it needs to be compiled, for that we need to know:\n",
    "* which optimizer we want to use (sgd is fine to begin with)\n",
    "* what is the loss (categorial_crossentropy for multiclass of the kind we have is the right choice)\n",
    "* which metrics to measure, accuracy is an okay choice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=\"adam\",loss=\"sparse_categorical_crossentropy\",metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A compiled model can be fitted on data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ginter/venv-jlab/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
      "\n",
      "Train on 22500 samples, validate on 2500 samples\n",
      "Epoch 1/5\n",
      "22500/22500 [==============================] - 74s 3ms/step - loss: 0.3191 - accuracy: 0.8652 - val_loss: 0.2971 - val_accuracy: 0.8824\n",
      "Epoch 2/5\n",
      "22500/22500 [==============================] - 81s 4ms/step - loss: 0.0966 - accuracy: 0.9668 - val_loss: 0.3840 - val_accuracy: 0.8804\n",
      "Epoch 3/5\n",
      "22500/22500 [==============================] - 74s 3ms/step - loss: 0.0386 - accuracy: 0.9872 - val_loss: 0.5524 - val_accuracy: 0.8760\n",
      "Epoch 4/5\n",
      "22500/22500 [==============================] - 76s 3ms/step - loss: 0.0191 - accuracy: 0.9944 - val_loss: 0.6906 - val_accuracy: 0.8688\n",
      "Epoch 5/5\n",
      "22500/22500 [==============================] - 85s 4ms/step - loss: 0.0167 - accuracy: 0.9945 - val_loss: 0.9025 - val_accuracy: 0.8632\n"
     ]
    }
   ],
   "source": [
    "hist=model.fit(feature_matrix,class_numbers,batch_size=100,verbose=1,epochs=5,validation_split=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.8823999762535095, 0.8804000020027161, 0.8759999871253967, 0.8687999844551086, 0.8632000088691711]\n"
     ]
    }
   ],
   "source": [
    "print(hist.history[\"val_accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We ran for 10 epochs of training\n",
    "* Made it to a decent accuracy on the validation data\n",
    "\n",
    "* But we do not have the model saved, so let's fix that and get the whole thing done\n",
    "* What constitutes a model (ie what we need to run the model on new data)\n",
    "  - The feature dictionary in the vectorizer\n",
    "  - The list of classes in their correct order\n",
    "  - The structure of the network\n",
    "  - The weights the network learned\n",
    "\n",
    "* Do all these things, and run again. This time we also increase the number of epochs to 100, see what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 22500 samples, validate on 2500 samples\n",
      "Epoch 1/100\n",
      "22500/22500 [==============================] - 93s 4ms/step - loss: 0.3162 - accuracy: 0.8706 - val_loss: 0.2945 - val_accuracy: 0.8884\n",
      "\n",
      "Epoch 00001: val_loss improved from inf to 0.29446, saving model to models/imdb_bow.weights.h5\n",
      "Epoch 2/100\n",
      "22500/22500 [==============================] - 86s 4ms/step - loss: 0.0961 - accuracy: 0.9660 - val_loss: 0.3727 - val_accuracy: 0.8744\n",
      "\n",
      "Epoch 00002: val_loss did not improve from 0.29446\n",
      "Epoch 3/100\n",
      "22500/22500 [==============================] - 79s 4ms/step - loss: 0.0367 - accuracy: 0.9884 - val_loss: 0.6194 - val_accuracy: 0.8596\n",
      "\n",
      "Epoch 00003: val_loss did not improve from 0.29446\n",
      "Restoring model weights from the end of the best epoch\n",
      "Epoch 00003: early stopping\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import Input, Dense\n",
    "from keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "import pickle\n",
    "import os\n",
    "\n",
    "def save_model(file_name,model,label_encoder,vectorizer):\n",
    "    \"\"\"Saves model structure and vocabularies\"\"\"\n",
    "    model_json = model.to_json()\n",
    "    with open(file_name+\".model.json\", \"w\") as f:\n",
    "        print(model_json,file=f)\n",
    "    with open(file_name+\".encoders.pickle\",\"wb\") as f:\n",
    "        pickle.dump((label_encoder,vectorizer),f)\n",
    "        \n",
    "example_count,feature_count=feature_matrix.shape\n",
    "class_count=len(label_encoder.classes_)\n",
    "\n",
    "inp=Input(shape=(feature_count,))\n",
    "hidden=Dense(200,activation=\"tanh\")(inp)\n",
    "outp=Dense(class_count,activation=\"softmax\")(hidden)\n",
    "model=Model(inputs=[inp], outputs=[outp])\n",
    "model.compile(optimizer=\"adam\",loss=\"sparse_categorical_crossentropy\",metrics=['accuracy'])\n",
    "\n",
    "# Save model and vocabularies, can be done before training\n",
    "save_model(\"models/imdb_bow\",model,label_encoder,vectorizer)\n",
    "# Callback function to save weights during training, if validation loss goes down\n",
    "save_cb=ModelCheckpoint(filepath=\"models/imdb_bow.weights.h5\", monitor='val_loss', verbose=1, save_best_only=True, mode='auto')\n",
    "# Callback to stop training when no improvement\n",
    "stop_cb=EarlyStopping(monitor='val_loss', patience=2, verbose=1, mode='auto', baseline=None, restore_best_weights=True)\n",
    "\n",
    "hist=model.fit(feature_matrix,class_numbers,batch_size=100,verbose=1,epochs=100,validation_split=0.1,callbacks=[save_cb,stop_cb])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network output= [[9.9930477e-01 6.9525826e-04]\n",
      " [7.9488242e-01 2.0511755e-01]\n",
      " [4.2360965e-02 9.5763910e-01]\n",
      " ...\n",
      " [6.8167770e-01 3.1832227e-01]\n",
      " [2.9446951e-01 7.0553046e-01]\n",
      " [1.4119088e-02 9.8588085e-01]]\n",
      "Maximum class for each example= [0 0 1 ... 0 1 1]\n",
      "Confusion matrix=\n",
      " [[1088  183]\n",
      " [  96 1133]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "         neg       0.92      0.86      0.89      1271\n",
      "         pos       0.86      0.92      0.89      1229\n",
      "\n",
      "    accuracy                           0.89      2500\n",
      "   macro avg       0.89      0.89      0.89      2500\n",
      "weighted avg       0.89      0.89      0.89      2500\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ginter/venv-jlab/lib/python3.7/site-packages/sklearn/preprocessing/_label.py:289: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    }
   ],
   "source": [
    "import numpy\n",
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "\n",
    "#Validation data used during training:\n",
    "val_instances,val_labels,_=hist.validation_data\n",
    "\n",
    "print(\"Network output=\",model.predict(val_instances))\n",
    "predictions=numpy.argmax(model.predict(val_instances),axis=1)\n",
    "print(\"Maximum class for each example=\",predictions)\n",
    "conf_matrix=confusion_matrix(list(val_labels),list(predictions))\n",
    "print(\"Confusion matrix=\\n\",conf_matrix)\n",
    "gold_labels=label_encoder.inverse_transform(list(val_labels))\n",
    "predicted_labels=label_encoder.inverse_transform(list(predictions))\n",
    "print(classification_report(gold_labels,predicted_labels))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning progress\n",
    "\n",
    "* The history object we get lets us inspect the accuracy during training\n",
    "* Remarks:\n",
    "  - Accuracy on training data keeps going up\n",
    "  - Accuracy on validation (test) data flattens out after a but over 10 epochs, we are learning very little past that point\n",
    "  - What we see is the network keeps overfitting on the training data to the end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXxU5fX48c9JSAgJkJAEBRK21gUCJCwRRFREZLMoBREBN3BBsWrFaqU/famltfIt1mpbiiKC0iJURS0KigsoWIsQVECgAkKEALKEfQmQ5Pz+uDfDTNaBzGSSyXm/Xnkxc587Myc3w7n3Ps89zxVVxRhjTPiKCHUAxhhjgssSvTHGhDlL9MYYE+Ys0RtjTJizRG+MMWHOEr0xxoS5ChO9iEwXkd0i8m0Z7SIifxGRTSKyWkQ6e7XdKiIb3Z9bAxm4McYY//hzRP8K0L+c9gHA+e7PGGAKgIgkAk8A3YCuwBMi0qgywRpjjDlzFSZ6VV0C7CtnlUHATHUsAxJEpCnQD/hIVfep6n7gI8rfYRhjjAmCOgF4jxRgm9fzHHdZWctLEJExOGcDxMXFdWnTpk0AwjLGmNpj5cqVe1W1cWltgUj0laaqU4GpAJmZmZqVlRXiiIwxpmYRkR/KagvEVTfbgeZez1PdZWUtN8YYU4UCkejnAbe4V99cDBxU1Z3AQqCviDRyB2H7usuMMcZUoQq7bkRkNnAFkCwiOThX0kQBqOoLwALgamATcAwY7bbtE5HfASvct5qgquUN6hpjjAmCChO9qo6ooF2BX5TRNh2YfnahnXbq1ClycnLIy8ur7FsZU0JMTAypqalERUWFOhRjgqJaDMZWJCcnhwYNGtCqVStEJNThmDCiquTm5pKTk0Pr1q1DHY4xQVEjpkDIy8sjKSnJkrwJOBEhKSnJzhZNWKsRiR6wJG+Cxr5bJtzVmERvjDHm7Fii90OvXr1YuND3ytDnnnuOsWPHlvu6+vXrA7Bjxw6GDh1a6jpXXHEFFRWIPffccxw7dszz/Oqrr+bAgQP+hB4w2dnZvPbaa1X6mcaYwLBE74cRI0YwZ84cn2Vz5sxhxIhyL0jyaNasGW+++eZZf37xRL9gwQISEhLO+v3ORnVJ9Pn5+aEOwZgaxxK9H4YOHcr8+fM5efIk4CS9HTt2cNlll3HkyBF69+5N586d6dChA//+979LvD47O5v27dsDcPz4cYYPH07btm0ZPHgwx48f96w3duxYMjMzadeuHU888QQAf/nLX9ixYwe9evWiV69eALRq1Yq9e/cC8Oyzz9K+fXvat2/Pc8895/m8tm3bcuedd9KuXTv69u3r8zlF3njjDdq3b09GRgaXX345AAUFBTz88MNcdNFFpKen8+KLLwIwfvx4li5dSseOHfnzn//s8z7lbYOZM2eSnp5ORkYGN998MwC7du1i8ODBZGRkkJGRwRdffOGzjQCeeeYZnnzyScA563nggQfIzMzk+eef591336Vbt2506tSJq666il27dnniGD16NB06dCA9PZ25c+cyffp0HnjgAc/7vvTSS4wbN66cv7Yx4adGXF7p7bfvrmXdjkMBfc+0Zg154pp2ZbYnJibStWtX3n//fQYNGsScOXMYNmwYIkJMTAxvv/02DRs2ZO/evVx88cVce+21ZQ7wTZkyhdjYWNavX8/q1avp3NkzfT9PPfUUiYmJFBQU0Lt3b1avXs3999/Ps88+y+LFi0lOTvZ5r5UrVzJjxgy+/PJLVJVu3brRs2dPGjVqxMaNG5k9ezYvvfQSw4YNY+7cudx0000+r58wYQILFy4kJSXF0xX08ssvEx8fz4oVKzhx4gQ9evSgb9++TJw4kWeeeYb33nuvxO9U1jZYt24dv//97/niiy9ITk5m3z6nXu7++++nZ8+evP322xQUFHDkyBH2799f7t/o5MmTni6u/fv3s2zZMkSEadOm8cc//pE//elP/O53vyM+Pp41a9Z41ouKiuKpp55i0qRJREVFMWPGDM/Oy5jaosYl+lAp6r4pSvQvv/wy4FyH/f/+3/9jyZIlREREsH37dnbt2kWTJk1KfZ8lS5Zw//33A5Cenk56erqn7fXXX2fq1Knk5+ezc+dO1q1b59Ne3Oeff87gwYOJi4sDYMiQISxdupRrr72W1q1b07FjRwC6dOlCdnZ2idf36NGDUaNGMWzYMIYMGQLAhx9+yOrVqz1dTQcPHmTjxo1ER0eXGUdZ22DRokVcf/31nh1UYmIiAIsWLWLmzJkAREZGEh8fX2Giv+GGGzyPc3JyuOGGG9i5cycnT570XP/+8ccf+3SxNWrk3P7gyiuv5L333qNt27acOnWKDh06lPtZxoSbGpfoyzvyDqZBgwYxbtw4vvrqK44dO0aXLl0AmDVrFnv27GHlypVERUXRqlWrs7ome8uWLTzzzDOsWLGCRo0aMWrUqEpd2123bl3P48jIyFK7bl544QW+/PJL5s+fT5cuXVi5ciWqyl//+lf69evns+6nn35a5mcFYhvUqVOHwsJCz/Piry/amQHcd999PPjgg1x77bV8+umnni6estxxxx384Q9/oE2bNowePfqM4jImHFgfvZ/q169Pr169uO2223wGYQ8ePMg555xDVFQUixcv5ocfypwpFIDLL7/cM6j57bffsnr1agAOHTpEXFwc8fHx7Nq1i/fff9/zmgYNGnD48OES73XZZZfxzjvvcOzYMY4ePcrbb7/NZZdd5vfv9P3339OtWzcmTJhA48aN2bZtG/369WPKlCmcOnUKgA0bNnD06NEyYyhvG1x55ZW88cYb5ObmAni6bnr37s2UKVMAZ0zg4MGDnHvuuezevZvc3FxOnDhRaheR9+elpDi3Nnj11Vc9y/v06cPkyZM9z4vOErp168a2bdt47bXX/B5AN6bKHD8A3y+CJZPgP38JykdYoj8DI0aMYNWqVT7J4sYbbyQrK4sOHTowc+ZMKrppytixYzly5Aht27bl8ccf95wZZGRk0KlTJ9q0acPIkSPp0aOH5zVjxoyhf//+nsHYIp07d2bUqFF07dqVbt26cccdd9CpUye/f5+HH36YDh060L59ey655BIyMjK44447SEtLo3PnzrRv35677rqL/Px80tPTiYyMJCMjo8RgbFnboF27djz66KP07NmTjIwMHnzwQQCef/55Fi9eTIcOHejSpQvr1q0jKiqKxx9/nK5du9KnT59yt+OTTz7J9ddfT5cuXXzGLR577DH279/vGWBevHixp23YsGH06NHD051jTEjkn4TtX8Hyl+Dtu+GvmfB/LeEfg2HR752EHwTizElWfZR245H169fTtm3bEEVkwsHAgQMZN24cvXv3LrXdvmMm4FRh32YnsW/Pgu0rYedqKDjhtMc1hpRMSO0CKV2gWWeod/aXTYvISlXNLK2txvXRG3MmDhw4QNeuXcnIyCgzyRsTEEdznWS+feXpxH7cvcggKhaadoSud0JqppPY45tDFU2/YYnehLWEhAQ2bNgQ6jBMuDl1HH5cAzlZpxP7/my3UeCcttBmoJPQUzOhcVuIDF26tURvjDHlKSyE3I1OQi9K7Lu+hUK3SrthCqR0hi6j3S6YjlC3QWhjLsYSvTHGeDu86/RRek4W7PgaTrhFmtENIKUTXHKf07+e0gUaNg1tvH6wRG+Mqb1OHoUd33j1q38FB7c5bRIJ57aDDkOdhJ6SCcnnQ0RkaGM+C5bojTG1Q2EB7PmfV7/6Sti9DtQt1EtoAakXwcVjncTeJB2iY0Mbc4BYovdDbm6u54qNH3/8kcjISBo3bgzA8uXLy50eoMjo0aMZP348F154YZnrTJ48mYSEBG688cbABO6nRYsWERsby8UXX1yln2tM0KjCoe1e/epfOV0wp4467THxTjK/8Gr3aL0L1G8c2piDyK9ELyL9geeBSGCaqk4s1t4S5ybgjYF9wE2qmuO2/RH4GU5x1kfAL7W6XbxfgaSkJL755hvAKdapX78+Dz30kM86qoqqEhFReg3ajBkzKvycX/yi1HusB92iRYtITk4OeaIvKCggMrLmnRabaiDvEOz4yk3s7tH6kR+dtshoaNIBOt14ul896adVdmljdVBhZayIRAKTgQFAGjBCRNKKrfYMMFNV04EJwNPuay8BegDpQHvgIqBnwKIPsU2bNpGWlsaNN95Iu3bt2LlzJ2PGjPFMNTxhwgTPupdeeinffPMN+fn5JCQkMH78eDIyMujevTu7d+8GnMrOoqmGL730UsaPH0/Xrl258MIL+eKLLwA4evQo1113HWlpaQwdOpTMzEzPTsjbww8/TFpaGunp6TzyyCOAMz3wkCFDyMzMpGvXrixbtozvv/+eadOmMWnSJDp27Oj5nCLLli2je/fudOrUiR49erBx40bAmRd+3LhxtG/fnvT0dP7+978D8OWXX9K9e3cyMjLo1q0bx44dY9q0aT5TBffv35/PP//csy0eeOAB0tPTWb58OU888QQXXXQR7du35+6776bomGDDhg1ceeWVZGRk0LlzZ7Kzsxk5cqTPVAk33HAD8+fPr9wf1VR/BaecfvUV0+Cde+BvXWFiC5g5CD6Z4HTP/KQnDPgj3LEIfpMDdy6CqydBxg2QfF6tSvLg3xF9V2CTqm4GEJE5wCBgndc6acCD7uPFwDvuYwVigGhAgChgV6Uifn+8c/1qIDXpAAMmVrxeKf73v/8xc+ZMMjOdgrSJEyeSmJhIfn4+vXr1YujQoaSl+e4XDx48SM+ePZk4cSIPPvgg06dPZ/z48SXeW1VZvnw58+bNY8KECXzwwQf89a9/pUmTJsydO5dVq1b5THNcZNeuXSxYsIC1a9ciIp4piO+//35+/etfc/HFF5Odnc3AgQP59ttvueOOO0hOTvZJxkXatm3L0qVLqVOnDh988AGPPfYY//rXv5gyZQo7duxg1apVREZGsm/fPvLy8hg+fDhz586lc+fOHDx40GdytdIcPHiQyy+/3LODu/DCC/ntb3+LqjJy5Eg++OADBgwYwIgRI3jyySe55ppryMvLo7CwkNtvv50pU6YwcOBA9u/fz4oVK6rFzVFMAKk616d7CpFWws5VkO9Oeheb5ByldxjqXOLYrDPEJoY05OrIn0SfAmzzep4DdCu2zipgCE73zmCggYgkqep/RWQxsBMn0f9NVdcX/wARGQOMAWjRosUZ/xKh9NOf/tST5AFmz57Nyy+/TH5+Pjt27GDdunUlEn29evUYMGAA4EwhvHTp0lLfu2jqYO9phj///HPPEXpGRgbt2pWczTMxMZGIiAjuvPNOfvaznzFw4EDAmcb3u+++86y3f//+Ume19HbgwAFuueUWvv/+e5/lH3/8MQ888ICnqyUxMZGvv/6aFi1aeHY+8fHx5b43QHR0NIMHD/Y8/+STT5g0aRJ5eXns3buXLl26cPHFF7N3716uueYawJn/HpxJ0+69915yc3OZPXs2w4YNs66fmu7YPqcLJseruvSYMykedWKc6tLM209PG5DQstYdnZ+NQA3GPgT8TURGAUuA7UCBiJwHtAVS3fU+EpHLVNUns6nqVGAqOHPdlPtJZ3nkHSze0+du3LiR559/nuXLl5OQkMBNN91U6nS93oO3kZGRZd4er+houLx1ShMVFUVWVhYfffQRb7zxBlOmTOHDDz/0nCH4M3hc5NFHH6Vfv37cc889bNq0if79+/v92iLlTUFcr149z01ajh07xr333stXX31FSkoKjz32WLnTHYsIN910E6+99hqvvvoqs2bNOuPYTAjln3DOzr0LkfYVHVAINL4QLhjgHKmnZsI5aRAZFdKQayp/Zq/cDjT3ep7qLvNQ1R2qOkRVOwGPussO4BzdL1PVI6p6BHgf6B6QyKuhQ4cO0aBBAxo2bMjOnTtL3FA8EHr06MHrr78OwJo1a1i3bl2JdQ4fPsyhQ4cYOHAgf/7zn/n6668BuOqqq3ym8S3q269oCuKiKYFfeeUVz/I+ffrwwgsvUFBQADhTEKelpbF161a++uorwNkeBQUFtGrViq+//hpVJTs7m5UrV5b6WcePHyciIoLk5GQOHz7M3LlzAecGIo0bN+bdd98FnB1F0T10R48ezaRJk6hbt265VzSZECsshL2bYNUcWPAwTO0Ff0iBab3h/V/DliXOtAG9n4Bb5sH4rfCLL+Hnk+Gi26FphiX5SvDniH4FcL6ItMZJ8MOBkd4riEgysE9VC4Hf4FyBA7AVuFNEnsbpuukJPBeg2Kudzp07k5aWRps2bWjZsqXPVMOBct9993HLLbeQlpbm+SneRXLw4EGGDBnCiRMnKCws5NlnnwWcyzfHjh3LjBkzPGMIkydPZtCgQVx//fW89dZbTJ48mUsuucTzXo888gi33XYbv/3tbz3dTQB33XUXGzduJD09nTp16jB27FjuvvtuZs+ezdixY8nLy6NevXosWrSInj17kpKSQtu2bWnXrp3nzlfFJSUlceutt5KWlkbTpk3p1u10D+GsWbO46667ePTRR4mOjmbu3Lm0bNmSZs2accEFFzB8+PBAbmZTWUf2+E7utX0l5B102qLinKP07vecvgomPiW08YY5v6YpFpGrcRJ0JDBdVZ8SkQlAlqrOE5GhOFfaKE7XzS9U9YR7xc7fgcvdtg9U9cHSP8Vh0xSXLz8/n/z8fGJiYti4cSN9+/Zl48aN1KlTO0sijh49SocOHVi1ahUNGpz9/CL2HauEk8ecAVLvxH5gq9MmEXBOu9PdLymZTpdMDawure4qPU2xqi4AFhRb9rjX4zeBN0t5XQFw1xlFa8p15MgRevfuTX5+PqrKiy++WGuT/MKFC7nzzjt5+OGHK5XkzRkoLIC9G3xnbdy1DtTpwiO+uXOEfpE7HW/TDIiOK/89TdDVzgxRgyUkJJTZx13b9OvXj61bt4Y6jPB2aIfvYOmOb+CkO55TN96Z4OvScaerSxucG9p4TalqTKJXVc/VGcYEUg0r1A6eE4fdCb6yTleYHt7htEVEQZP2TsGRp7r0PCijEtxULzUi0cfExJCbm0tSUpIlexNQqkpubq7n2vxaoyDfmdDLe9bG3etxhtKARq2hVY/TszY26QBRtWwbhZEakehTU1PJyclhz549oQ7FhKGYmBhSU1MrXrGmUnWm3vWetXHHN5DvFsvVS3QSetqg010wVl0aVmpEoo+KiqJ169ahDsOYmuH4Aa/qUveI/ah7kBRZ1xkg7TLKvQqms3P0bmfKYa1GJHpjTBnyT8KuNU7XS9ERe+7G0+3JF8B5fbyqS9tBHf8ro014sERvTE2hCvs2n+5+ycmCH1dDwUmnPe4cJ5lnDHe7YDo7866bWs8SvTHV1dHcktWlx/c7bVGxzgRf3e46PWAan2pdMKZUluiNqQ5OHYedq30T+/5sp00ioHFbaDPQ7Vfv4jyPtP++xj/2TTGmqhUWOv3oPtWla6HQnaG0YYqTzLuMdqtLO0Ld+qGN2dRoluiNCbbDu7yKkLKce5eeOOS0RTdwqksvuf/0pY0Nm4Y2XhN2LNEbE0gnjrgTfHlVlx7Kcdoi6sC57dy7IbldMMkXWHWpCTpL9MacrcICp5rUp7p0Hah7k5WEltCiG6S40/E2TYeoeqGN2dRKluiN8dfRvZD9uW916amjTltMgnOE3uZnp7tg4pJDG68xLkv0xvgj+z8wewScOAiR0dAkHTrddPoqmMSf2KWNptqyRG9MRda/B2/eBo1awo1vQLOOUKduqKMyxm+W6I0pz8pX4L1x0Kyzk+Rtsi9TA1miN6Y0qrDkGVj8e2eumGGv2p2STI1lid6Y4goL4YNHYPlUSL8BBk2GyKhQR2XMWbNEb4y3/BPw9l2w9m3ofi/0+Z1d525qPL++wSLSX0S+E5FNIjK+lPaWIvKJiKwWkU9FJNWrrYWIfCgi60VknYi0Clz4xgTQicMw63onyfeZAP2esiRvwkKF32IRiQQmAwOANGCEiKQVW+0ZYKaqpgMTgKe92mYCk1S1LdAV2B2IwI0JqCN74JWBznXyP58CPX4Z6oiMCRh/Dle6AptUdbOqngTmAIOKrZMGLHIfLy5qd3cIdVT1IwBVPaKqxwISuTGBsj8bpveFPd/BiNnQcWSoIzImoPxJ9CnANq/nOe4yb6uAIe7jwUADEUkCLgAOiMhbIvK1iExyzxB8iMgYEckSkSy7L6ypUj+ugZf7wrF9cOs8uKBfqCMyJuAC1QH5ENBTRL4GegLbgQKcwd7L3PaLgJ8Ao4q/WFWnqmqmqmY2btw4QCEZU4Hsz2HG1c5kY7cthOZdQx2RMUHhT6LfDjT3ep7qLvNQ1R2qOkRVOwGPussO4Bz9f+N2++QD7wCdAxK5MZWx/l34xxBo0BRu/xDOaRPqiIwJGn8S/QrgfBFpLSLRwHBgnvcKIpIsIkXv9RtgutdrE0Sk6DD9SmBd5cM2phJWvgKv3+LMJnnbB84t+IwJYxUmevdI/F5gIbAeeF1V14rIBBG51l3tCuA7EdkAnAs85b62AKfb5hMRWQMI8FLAfwtj/KEKn02Cd38JP+0Nt/zbpjQwtYKoaqhj8JGZmalZWVmhDsOEm8JCeP/XsOIlSB8Og/5m1a4mrIjISlXNLK3NKmNN+POudr3kPrhqghVCmVrFEr0JbycOw5wbYctnznQGPe4PdUTGVDlL9CZ8HdkDs66DH7+Fn78AHUeEOiJjQsISvQlP+7PhH4Ph0E6n2tUKoUwtZonehJ8f18A/r3P65m+dZ4VQptazESkTXqza1ZgSLNGb8GHVrsaUyhK9CQ9ZM6za1ZgyWB+9qdlUYckkWPyU3dvVmDJYojc1V2EBvP+IVbsaUwFL9KZmsmpXY/xmid7UPFbtaswZsURvahardjXmjFmiNzWHT7XrHLigb6gjMqZGsERvagardjXmrNnolan+rNrVmEqxRG+qt3XzrNrVmEqyRG+qr6wZ8MatVu1qTCVZH72pfryrXc/vC9e/YtWuxlSCJXpTvXhXu2aMgGv/atWuxlSSX103ItJfRL4TkU0iMr6U9pYi8omIrBaRT0UktVh7QxHJEZG/BSpwE4byT8Dc250kf8l9MOjvluSNCYAKE72IRAKTgQFAGjBCRNKKrfYMMFNV04EJwNPF2n8HLKl8uCZs5R2CWUOdKQ36/A76/t6mNDAmQPz5n9QV2KSqm1X1JDAHGFRsnTRgkft4sXe7iHQBzgU+rHy4Jiwd2Q2vDoTs/zjVrjalgTEB5U+iTwG2eT3PcZd5WwUMcR8PBhqISJKIRAB/Ah4q7wNEZIyIZIlI1p49e/yL3ISHfVtgej/Ys8GpdrUpDYwJuECdGz8E9BSRr4GewHagALgHWKCqOeW9WFWnqmqmqmY2btw4QCGZam/naifJH98Pt75rUxoYEyT+XHWzHWju9TzVXeahqjtwj+hFpD5wnaoeEJHuwGUicg9QH4gWkSOqWmJA19QyW5bCnJFQt4GT5BtfGOqIjAlb/iT6FcD5ItIaJ8EPB0Z6ryAiycA+VS0EfgNMB1DVG73WGQVkWpI3rJsHc++ARq3g5resEMqYIKuw60ZV84F7gYXAeuB1VV0rIhNE5Fp3tSuA70RkA87A61NBitfUdJ5q1wyrdjWmioiqhjoGH5mZmZqVlRXqMEyglah2fRWiY0MdlTFhQ0RWqmpmaW1WGWuCz6pdjQkpS/QmuPJPwFtjYN07cMn90GcCiIQ6KmNqFUv0JnjyDsG/boQtS5xK10vuC3VExtRKluhNcBzZ7UxpsGstDH4RMoaHOiJjai1L9Cbw9m1x7u16ZJdT7Xp+n1BHZEytZoneBNbO1c6RfMFJuGUeNL8o1BEZU+vZ9IAmcLYshVd+BhFR7r1dLckbUx1YojeBsW4e/LPo3q4LbUoDY6oRS/Sm8rKmu9WuHa3a1ZhqyProzdlThc/+CJ/+Ac7v597b1apdjaluLNGbs1NYAO//GlZMg4yRcO1frNrVmGrKEr05c1btakyNYonenBmrdjWmxrFEb/xn1a7G1EiW6I1/rNrVmBrLEr2p2M7V8M/roPCUVbsaUwPZdfSmfEXVrpHRVu1qTA1lid6Ubd2/nWrXhs2s2tWYGswSvSld1nR4/VZo1glGv2/VrsbUYNZHb3xZtasxYcevI3oR6S8i34nIJhEZX0p7SxH5RERWi8inIpLqLu8oIv8VkbVu2w2B/gVMABUWwIKHnCSfMRKGz7Ikb0wYqDDRi0gkMBkYAKQBI0QkrdhqzwAzVTUdmAA87S4/Btyiqu2A/sBzIpIQqOBNAOWfgDdHO1Ma9Pgl/PzvNqWBMWHCnyP6rsAmVd2sqieBOcCgYuukAYvcx4uL2lV1g6pudB/vAHYDjQMRuAmgvENOIdS6f0Pfp2xKA2PCjD+JPgXY5vU8x13mbRUwxH08GGggIkneK4hIVyAa+L74B4jIGBHJEpGsPXv2+Bu7CYQju53LJ3/4AgZPhUvuDXVExpgAC9RVNw8BPUXka6AnsB0oKGoUkabAP4DRqlpY/MWqOlVVM1U1s3FjO+CvMvu2wMt9IXeTU+2aYUMoxoQjf6662Q4093qe6i7zcLtlhgCISH3gOlU94D5vCMwHHlXVZYEI2gSAVbsaU2v4c0S/AjhfRFqLSDQwHJjnvYKIJItI0Xv9BpjuLo8G3sYZqH0zcGGbStmyFGZcbdWuxtQSFSZ6Vc0H7gUWAuuB11V1rYhMEJFr3dWuAL4TkQ3AucBT7vJhwOXAKBH5xv3pGOhfwpyBomrX+BS4/UOrdjWmFhBVDXUMPjIzMzUrKyvUYYSnFS/D/F9B865On3xsYqgjMsYEiIisVNXM0tqsMrY2UIXP/g8+fdqqXY2phSzRhzu7t6sxtZ4l+nCWfwLeutPpl+/xS7jqt1YIZUwtZIk+XOUdgjkjIXupU+1qhVDG1FqW6MPRkd3ONfK71znVrlYIZUytZok+3OzbDP8YYvd2NcZ4WKIPJztXwT+HQmE+3PoupJZ6pZUxppaxO0yFiy1LYIbXvV0tyRtjXJbow8Had5w+eU+16wWhjsgYU41Yoq/pVrwMb4zyurdr8RmkjTG1nfXR11Te1a4X9IehM6za1RhTKkv0NVFhASx4GLJeho43whZ2Ij0AABN8SURBVDXPW7WrMaZMluhrGqt2NcacIUv0NYlVuxpjzoIl+prCql2NMWfJEn1N4FPt+i84/6pQR2SMqUEs0Vd3Vu1qjKkku46+OrNqV2NMAFiir6481a6pVu1qjKkUS/TVkU+16wKrdjXGVIpfiV5E+ovIdyKySUTGl9LeUkQ+EZHVIvKpiKR6td0qIhvdn1sDGXzYUYXFT8P8B+GCfnDzO3YDb2NMpVWY6EUkEpgMDADSgBEiklZstWeAmaqaDkwAnnZfmwg8AXQDugJPiEijwIUfRgoLYP6v4LOJTrXrDbNsSgNjTED4c0TfFdikqptV9SQwBxhUbJ00YJH7eLFXez/gI1Xdp6r7gY+A/pUPO8ycynO6arJehh4PwKDJEGkXRBljAsOfRJ8CbPN6nuMu87YKGOI+Hgw0EJEkP1+LiIwRkSwRydqzZ4+/sYeHvEMwayisnwf9/gB9bEoDY0xgBWow9iGgp4h8DfQEtgMF/r5YVaeqaqaqZjZu3DhAIdUAh3fBK1fD1v/CkJeg+y9CHZExJgz50z+wHWju9TzVXeahqjtwj+hFpD5wnaoeEJHtwBXFXvtpJeINH/s2wz8GO1MbWLWrMSaI/DmiXwGcLyKtRSQaGA7M815BRJJFpOi9fgNMdx8vBPqKSCN3ELavu6x227kKXu7ndNvc+q4leWNMUFWY6FU1H7gXJ0GvB15X1bUiMkFErnVXuwL4TkQ2AOcCT7mv3Qf8DmdnsQKY4C6rvYqqXevUtWpXY0yVEFUNdQw+MjMzNSsrK9RhBMfad5y55BN/CjfNtUIoY0zAiMhKVS31yNEqY6uKVbsaY0LELtYONlX4dKJTCGX3djXGhIAl+mAqLIAFD0HWdOh4k3tvV9vkxpiqZVknWE7lOf3x6+c51a5XPWmFUMaYkLBEHwx5B2HOjc69Xfv9wQqhjDEhZYk+0A7vglnXwe71TrVr+rBQR2SMqeUs0QeSVbsaY6ohS/SBsnOVc0eowgK7t6sxplqx6+gDYfNnbrVrjFW7GmOqHUv0lbX2HWeaYbu3qzGmmgqbrpuT+YU8/OYqWiTG0jwxlhbuT5OGMUREBOmyxhXTYP5D0LwbjJwD9ezmWcaY6idsEv3+YydZ+cN+3l21g0Kv6XuiIyNITaznSfzFdwRxdc9iE/hUuw6AodOt2tUYU22FTaI/t2EMnz9yJacKCtlx4Dhb9x07/ZPr/Lsyez+HT+T7vC65frRP4q/wbMCqXY0xNUzYZaioyAhaJsXRMimuRJuqcvD4qdJ3An6cDbSOj2TAxidosn0hJ7v/kui+dts/Y0z1F3aJvjwiQkJsNAmx0aSnJpRoL+9s4Lvs7Txb+EeaRK5jwqmbmb64G8krPi71bKBlUiznNgji2IAxxpyBWpXoK1Lm2cDhXeis38DuDWy9/Hk6J/Yl+QzPBop2BC2TYmne6CzHBowx5ixYtqmIW+0qR/bAyH/R4ryraFHKaoEaG7CzAWNMoFmiL0+JatcuZa4azLEBOxswxlSGZYyybP7MmYGyXgLc9FalCqEqMzbg79lAy8RYWtjZgDGmFJboS1N0b9ek85x7uzZsFtSPC/bZQAt3J2BnA8bUTn79rxeR/sDzQCQwTVUnFmtvAbwKJLjrjFfVBSISBUwDOrufNVNVnw5g/IFXzapdg3024L0jsLMBY8JThYleRCKByUAfIAdYISLzVHWd12qPAa+r6hQRSQMWAK2A64G6qtpBRGKBdSIyW1WzA/x7VJ4qfPo0fPZ/Nara1c4GjDEV8ed/bldgk6puBhCROcAgwDvRK9DQfRwP7PBaHicidYB6wEngUADiDqzCApj/K1g5AzrdBAPDo9rVzgaMMeBfok8Btnk9zwG6FVvnSeBDEbkPiAOK7rjxJs5OYScQC4xT1X3FP0BExgBjAFq0KO3ixSA6lQdv3QHr34VLx0HvJ2pNtWuwzgZaek8lYWcDxoRcoP73jQBeUdU/iUh34B8i0h7nbKAAaAY0ApaKyMdFZwdFVHUqMBUgMzNTqSo+93Z9GrrfU2UfXd0F82yg+I7AzgaMCS5/Ev12oLnX81R3mbfbgf4AqvpfEYkBkoGRwAeqegrYLSL/ATKBzYSaz71dp0H69aGOqEaxswFjag5//getAM4XkdY4CX44TgL3thXoDbwiIm2BGGCPu/xKnCP8OOBi4LkAxX72cr+Hfw4Bt9qV8+zeroEUrLOBlsVnGLWzAWP8UmGiV9V8EbkXWIhz6eR0VV0rIhOALFWdB/wKeElExuEMwI5SVRWRycAMEVkLCDBDVVcH7bfxx45vnDtC+VHtaoLjbM8Gsn7Yz7ziZwN1IkhtVK/UnYCdDRjjENWq6xL3R2ZmpmZlZQXnzb2rXW9+G5LPD87nmKAp72xga+6xUs8GSr3pjJ0NmDAjIitVtdQbVteew521b8NbY6qs2tUER6DPBpo3qlfqTsDOBkw4qR3f5OUvwYKHq021qwmOyowNZJUxNlByYrk4WiXF0rhBXaSWXIZrar7wTvTe1a4XXu1Uu0bVC3VUJkQCeTYQGx3pSfo+/yZbl5CpfsI30YdptasJDn/OBrbvP84P+47xQ+5Rsvc6/27YdZhP1u/mZEGhZ926dSJoWXwH4O4EmsbXI9J2AqaKhWfm86l2fRB6P15rql1NcERFRtAqOY5WyXFAY5+2gkJl58Hj/JB7jOzco2TvPUp2rrMjWLJhDyfyT+8EoiMjaJ5Yj1bumUWr5NM7hJSEetSJjKji38zUBuGX6L2rXftPhIvHhjoiE+YiI4TURrGkNoqlx3nJPm2Fhcquw3meM4CiHUB27jH+uzmXYycLPOvWiRDnUtGkOFonx9EyKdbdITjvHV3HdgLm7IRXoj/8I/xzKOyxaldTPURECE3j69E0vh7df5rk06aq7Dlyguy9zpmA945g5Q/7OeI1OBwhkNKonifxe84IkpxB4pioyKr+1UwNEj6J/sBWeGUgHN0LI1+H83qHOiJjyiUinNMghnMaxNC1daJPm6qy7+hJnzOAH9xuoXnf7OBQXr7X+0Cz+HolxwWSY2mZGEe9aNsJ1Hbhk+hjk+HcdnDZQ1btamo8ESGpfl2S6telS8uSlwMfOOa1E/B0Cx3lw7U/knv0pM+65zas67MDKOoWapkUR32rFagValdlrDG1wKG8U/xQSndQdu4x9hw+4bNucv26vpeIJp/eIcTXiwrRb2DOhlXGGlOLNIyJokNqPB1S40u0HT2Rzw/FuoO27D3KfzbtZe5XeT7rJsZF+wwIe/+bEBtlBWM1iCV6Y2qRuLp1SGvWkLRmDUu0HT9ZwNZ9Jc8Elm/ZxzvfbMf75L9hTB1aJZ8eEG7ldaloUly07QSqGUv0xhgA6kVHcmGTBlzYpEGJthP5BWzbd9ytETjqqRlYte0A81f7Vg3Xr1un9DOB5DjOsakjQsISvTGmQnXrRHLeOfU575z6JdpO5hey/cBxZwfgVSy2fuchFq79kXyvvUC9qMjTyT/Zd2fQpKFNHREsluiNMZUSXSeC1snO1Txc6NuWX1DIjgN5JbqDNu05wqL/+U4dEV0ngpaJvgPDrd0dQbMEmzqiMizRG2OCpk5kBC2SnKmfS5s64sdDefyw9yhbirqD9jr/Lt3oO3VEVKTQPLH0geGURvWIsqkjymWJ3hgTEpERQkpCPVIS6nFJKVNH7D58ouQlonuPsazY1BGR3lNHeBeLJcXR3KaOACzRG2OqoYgIoUl8DE3iY7j4J6VPHeF9BlA0QPz1D773FYgQaJZQfOoIZ2C4RS2aOsISvTGmRvGeOuKiViWnjth/7JTnTGCL12Ry89fs5MCxU17vA00bxpSYRbSluzOIjQ6f9Bg+v4kxptYTERLjokmMi6Zzi9KnjvA+Ayj696N1u9h7xHfqiHMa1PU5Ayh63DIplgYxNatq2K9ELyL9geeBSGCaqk4s1t4CeBVIcNcZr6oL3LZ04EWgIVAIXKSqviV4xhhTBYpuLpPRvOTNZQ7lnWKr907A7Rb6bMMe3liZ47Nucv1oz5G/d7dQq6Q44mOr306gwrluRCQS2AD0AXKAFcAIVV3ntc5U4GtVnSIiacACVW0lInWAr4CbVXWViCQBB1S1oOQnOWyuG2NMdXP0RD5b953uBvIuHNt50Pe4NSE2yk36scW6heJoFMSpIyo7101XYJOqbnbfbA4wCFjntY7iHLEDxAM73Md9gdWqugpAVXPPPHxjjAmtuLp1aNu0IW2blpw6Iu+UO3VEsYHhrB/28+9VO3ymjmgQU6fEwHBrdyqJ5PrBmzrCn0SfAmzzep4DdCu2zpPAhyJyHxAHXOUuvwBQEVmIcxHtHFX9Y/EPEJExwBiAFi1anEn8xhgTUjFRkVxwbgMuOLfsqSOKzyK6ZvtB3v/2Rwq8qobjoiO59PxkXry51IPySgnUYOwI4BVV/ZOIdAf+ISLt3fe/FLgIOAZ84p5efOL9YlWdCkwFp+smQDEZY0xIlTd1RNEN570HhoM1NbQ/iX470Nzreaq7zNvtQH8AVf2viMQAyThH/0tUdS+AiCwAOgOfYIwxtZjvDeeDy5+SsRXA+SLSWkSigeHAvGLrbAV6A4hIWyAG2AMsBDqISKw7MNsT3759Y4wxQVbhEb2q5ovIvThJOxKYrqprRWQCkKWq84BfAS+JyDicgdlR6lzOs19EnsXZWSjO1Tjzg/XLGGOMKcluJWiMMWGgvMsrbbYfY4wJc5bojTEmzFmiN8aYMGeJ3hhjwpwlemOMCXOW6I0xJsxZojfGmDBnid4YY8KcJXpjjAlz1a4yVkT2AD9U4i2Sgb0BCieQLK4zY3GdGYvrzIRjXC1VtXFpDdUu0VeWiGSVVQYcShbXmbG4zozFdWZqW1zWdWOMMWHOEr0xxoS5cEz0U0MdQBksrjNjcZ0Zi+vM1Kq4wq6P3hhjjK9wPKI3xhjjxRK9McaEuRqT6EWkv4h8JyKbRGR8Ke11ReRfbvuXItLKq+037vLvRKRfFcf1oIisE5HVIvKJiLT0aisQkW/cn+L34Q12XKNEZI/X59/h1XariGx0f26t4rj+7BXTBhE54NUWzO01XUR2i8i3ZbSLiPzFjXu1iHT2agvm9qoorhvdeNaIyBcikuHVlu0u/0ZEAnrbNj/iukJEDnr9vR73aiv3OxDkuB72iulb9zuV6LYFc3s1F5HFbi5YKyK/LGWd4H3HVLXa/+Dcq/Z74CdANLAKSCu2zj3AC+7j4cC/3Mdp7vp1gdbu+0RWYVy9gFj38diiuNznR0K4vUYBfyvltYnAZvffRu7jRlUVV7H178O5R3FQt5f73pcDnYFvy2i/GngfEOBi4Mtgby8/47qk6POAAUVxuc+zgeQQba8rgPcq+x0IdFzF1r0GWFRF26sp0Nl93ADYUMr/yaB9x2rKEX1XYJOqblbVk8AcYFCxdQYBr7qP3wR6i4i4y+eo6glV3QJsct+vSuJS1cWqesx9ugxIDdBnVyqucvQDPlLVfaq6H/gI6B+iuEYAswP02eVS1SXAvnJWGQTMVMcyIEFEmhLc7VVhXKr6hfu5UHXfL3+2V1kq890MdFxV+f3aqapfuY8PA+uBlGKrBe07VlMSfQqwzet5DiU3kmcdVc0HDgJJfr42mHF5ux1nj10kRkSyRGSZiPw8QDGdSVzXuaeIb4pI8zN8bTDjwu3iag0s8locrO3lj7JiD+b2OlPFv18KfCgiK0VkTAji6S4iq0TkfRFp5y6rFttLRGJxkuVcr8VVsr3E6VbuBHxZrClo37E6ZxqkOTsichOQCfT0WtxSVbeLyE+ARSKyRlW/r6KQ3gVmq+oJEbkL52zoyir6bH8MB95U1QKvZaHcXtWaiPTCSfSXei2+1N1e5wAficj/3CPeqvAVzt/riIhcDbwDnF9Fn+2Pa4D/qKr30X/Qt5eI1MfZuTygqocC+d7lqSlH9NuB5l7PU91lpa4jInWAeCDXz9cGMy5E5CrgUeBaVT1RtFxVt7v/bgY+xdnLV0lcqprrFcs0oIu/rw1mXF6GU+y0Oojbyx9lxR7M7eUXEUnH+RsOUtXcouVe22s38DaB67KskKoeUtUj7uMFQJSIJFMNtpervO9XULaXiEThJPlZqvpWKasE7zsWjIGHQP/gnHlsxjmVLxrAaVdsnV/gOxj7uvu4Hb6DsZsJ3GCsP3F1whl8Or/Y8kZAXfdxMrCRAA1K+RlXU6/Hg4FlenrgZ4sbXyP3cWJVxeWu1wZnYEyqYnt5fUYryh5c/Bm+A2XLg729/IyrBc640yXFlscBDbwefwH0r8K4mhT9/XAS5lZ32/n1HQhWXG57PE4/flxVbS/3d58JPFfOOkH7jgVs4wb7B2dEegNO0nzUXTYB5ygZIAZ4w/3SLwd+4vXaR93XfQcMqOK4PgZ2Ad+4P/Pc5ZcAa9wv+hrg9iqO62lgrfv5i4E2Xq+9zd2Om4DRVRmX+/xJYGKx1wV7e80GdgKncPpAbwfuBu522wWY7Ma9Bsisou1VUVzTgP1e368sd/lP3G21yv07P1rFcd3r9f1ahteOqLTvQFXF5a4zCucCDe/XBXt7XYozBrDa6291dVV9x2wKBGOMCXM1pY/eGGPMWbJEb4wxYc4SvTHGhDlL9MYYE+Ys0RtjTJizRG+MMWHOEr0xxoS5/w8GIU09ccwqGwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.ylim(0.85,1.0)\n",
    "plt.plot(hist.history[\"val_accuracy\"],label=\"Validation set accuracy\")\n",
    "plt.plot(hist.history[\"accuracy\"],label=\"Training set accuracy\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary\n",
    "\n",
    "* We put together a program to train a neural network classifier for sentiment detector\n",
    "* We learned the necessary code/techniques to save models, and feed the training with data in just the right format\n",
    "* We observed the training across epochs\n",
    "* We saw how the classifier can be applied to various text classification problems\n",
    "* The IMDB sentiment classifier ended up at nearly 90% accuracy, the state of the art is about 95%, we got surprisingly far in few lines of code\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}