bow_classifier_with_embeddings_simpler.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bag-of-words classifier with pretrained word embeddings\n",
    "\n",
    "- During the lecture we will cover the concept of embeddings and the simple word2vec method\n",
    "- If we have a trained word embeddings model, we can transfer that knowledge into a new task and model (transfer learning)\n",
    "- What we achieve here: Initialize the weights in the classifier with pretrained word embeddings\n",
    "- Word embeddings downloaded at: https://dl.fbaipublicfiles.com/fasttext/vectors-english/wiki-news-300d-1M.vec.zip\n",
    "\n",
    "### Read data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%script bash\n",
    "\n",
    "#mkdir -p data\n",
    "#cd data\n",
    "#wget --quiet https://dl.fbaipublicfiles.com/fasttext/vectors-english/wiki-news-300d-1M.vec.zip\n",
    "#unzip wiki-news-300d-1M.vec.zip\n",
    "#wget https://github.com/TurkuNLP/Deep_Learning_in_LangTech_course/raw/master/data/imdb_train.json\n",
    "#cd .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'class': 'pos', 'text': \"This movie was absolutely wonderful. The pre-partition time and culture has been recreated beautifully. Urmila has given yet another brilliant performance. What I truly admire about this movie is that it doesn't resort to Pakistan-bashing that is running rampant in movies like Gadar and LOC. With the partition as a backdrop, the movie does not divert to political issues or focus on violence or what is right and wrong. The movie always centers around the tragic story of Urmila's life. Her fragile relationship with Manoj Bajpai has been depicted excellently. The movie actually shows how the people, both Hindus and Muslims, have suffered from this partition. The theme that there is only one religion is truly prevalent in this film.\"}\n",
      "[\"This movie was absolutely wonderful. The pre-partition time and culture has been recreated beautifully. Urmila has given yet another brilliant performance. What I truly admire about this movie is that it doesn't resort to Pakistan-bashing that is running rampant in movies like Gadar and LOC. With the partition as a backdrop, the movie does not divert to political issues or focus on violence or what is right and wrong. The movie always centers around the tragic story of Urmila's life. Her fragile relationship with Manoj Bajpai has been depicted excellently. The movie actually shows how the people, both Hindus and Muslims, have suffered from this partition. The theme that there is only one religion is truly prevalent in this film.\", \"La Sanguisuga Conduce la Danza, or The Bloodsucker Leads the Dance as I believe is it's common English title, is set on 'Ireland' in '1902' where the mysterious Count Richard Marnack (Giacomo Rossi-Stuart) invites soon-to-be out of work theatre actress Evelyn (Patrizia Webley as Patrizia De Rossi) & three more of her fellow soon-to-be out of work actress friends Cora (Krista Nell), Penny (Lidia Olizzi) & Rosalind (Marzia Damon as Caterina Chiani) to his castle situated on a small island just off the coast. At first Evelyn is reluctant but is persuaded when it is agreed stage hand Samuel (Leo Valeriano) goes along as well. Once there Count Marnack tells his guests that his Father & Grandfather both cut the heads off their cheating wives using a ceremonial knife & there's a feeling of unease when it turns out that Evelyn looks EXACTLY the same as the current Count's wife who ran away not too long ago... Along with having to worry about weirdo servants it turns out that someone wants to use the knife themselves to chop a few heads off...  This Italian production was written & directed by Alfredo Rizzo & is total, complete & utter crap from start to finish. First things first lets start the criticism with the title The Bloodsucker Leads the Dance, lets examine that title because when I do I feel somewhat cheated that there aren't any Vampires or any form of bloodsucking whatsoever, no-one 'leads' anything at anytime & there most certainly isn't any dance or dancing so don't expect any of these things. What you should expect though is a tedious, dull, boring pointless little 'giallo' that takes over an hour before anyone is killed & any sort of murder mystery starts to take shape, the first hour of La Sanguisuga Conduce la Danza is as plot less & uninteresting as anything out there. The best way to imagine it is to think of the most boring soft-core porn film you've ever seen, then cut most of the soft-core porn out & that just leaves bad actors, bad dubbing, bad dialogue, a tiny bit of soft-core sex & lesbianism & absolutely nothing else. Yep, it really is that bad. The mystery elements are crap, people do illogical things for no apparent reason & the overly complicated 'twist' ending is as bad as the rest of it, The Sixth Sense (1999) this ain't!  Director Rozzi does an OK job & La Sanguisga Conduce la Danza actually has a nice look & feel to it despite it's obvious low budget, I mean there are scenes where he shows a 'storm' raging outside the castle however the stock footage used is in black & white! So the film jumps from bright colour to black & white whenever it cuts to the storm & back to colour again! There is one absolutely hilarious scene where a maid & her friend discuss her breasts & she lets her friend feel them who is then full of compliments, this scene is so funny & just so unnatural that it's priceless & easily the films best moment even if a bad 70's porno would be embarrassed by the dialogue! There is minimal nudity & the sex/lesbian scenes are very tame. Forget about any blood or gore as there isn't any apart from a decapitated head.  Technically the films pretty good, the period setting & production design are actually quite impressive although it's not massive in scope it does the job effectively enough. The film has a nice sense of colour & the cinematography is fine. The film is dubbed so it's hard to give an opinion on the original performances but the voice actors are terrible & the dialogue is even worse than usual.  La Sanguisuga Conduce la Danza is a terrible film, it makes no sense, it has virtually no story for over an hour, it has possibly the most misleading title ever & is really dull & boring with a confused stupid 'twist' ending. As far as Euro horror fans go there are plenty of much better films than this so please don't waste your time, as for anyone else this is definitely one to avoid. Trivia Note: the notorious (& banned in the UK) Nazi exploitation/horror film Horrifying Experiments of the S.S Last Days (1977) AKA SS Hell Camp & The Beast in Heat was directed by Luigi Batzella & he has a fairly substantial role as a police detective in La Sanguisuga Conduce la Danza.\"]\n",
      "['pos', 'neg']\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import random\n",
    "with open(\"data/imdb_train.json\") as f:\n",
    "    data=json.load(f)\n",
    "random.shuffle(data) \n",
    "print(data[0])\n",
    "\n",
    "# We need to gather the texts, into a list\n",
    "texts=[one_example[\"text\"] for one_example in data]\n",
    "labels=[one_example[\"class\"] for one_example in data]\n",
    "print(texts[:2])\n",
    "print(labels[:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use gensim to read the embedding model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import KeyedVectors\n",
    "\n",
    "#Only grab the 100K most common entries\n",
    "vector_model=KeyedVectors.load_word2vec_format(\"data/wiki-news-300d-1M.vec.gz\", binary=False, limit=100000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with the embeddings\n",
    "\n",
    "* `vector_model.vocab`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Words from embedding model: 100000\n",
      "First 50 words: [',', 'the', '.', 'and', 'of', 'to', 'in', 'a', '\"', ':', ')', 'that', '(', 'is', 'for', 'on', '*', 'with', 'as', 'it', 'The', 'or', 'was', \"'\", \"'s\", 'by', 'from', 'at', 'I', 'this', 'you', '/', 'are', '=', 'not', '-', 'have', '?', 'be', 'which', ';', 'all', 'his', 'has', 'one', 'their', 'about', 'but', 'an', '|']\n"
     ]
    }
   ],
   "source": [
    "# sort based on the index to make sure they are in the correct order\n",
    "words=[k for k,v in sorted(vector_model.vocab.items(), key=lambda x:x[1].index)]\n",
    "print(\"Words from embedding model:\",len(words))\n",
    "print(\"First 50 words:\",words[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalize the vectors\n",
    "\n",
    "- Easier to learn on top of these vectors when the magnitude does not vary much"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before normalization: [-0.0234 -0.0268 -0.0838  0.0386 -0.0321  0.0628  0.0281 -0.0252  0.0269\n",
      " -0.0063]\n",
      "After normalization: [-0.0163762  -0.01875564 -0.05864638  0.02701372 -0.02246478  0.04394979\n",
      "  0.01966543 -0.0176359   0.01882563 -0.00440898]\n"
     ]
    }
   ],
   "source": [
    "print(\"Before normalization:\",vector_model.get_vector(\"in\")[:10])\n",
    "vector_model.init_sims(replace=True)\n",
    "print(\"After normalization:\",vector_model.get_vector(\"in\")[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text analyzer and vectorizer\n",
    "\n",
    "- When we use an embedding layer (keras.layers.Embedding) the input data must be a sequence, not a bag-of-words vector\n",
    "- This prepares us for working with sequences, but we must give up on our trusty `CountVectorizer`\n",
    "- You can use CountVectorizer only as an analyzer without building the feature matrix\n",
    "- We will have to build the vectorizer part later ourselves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['have', 'dog']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "import numpy\n",
    "\n",
    "vectorizer=CountVectorizer(analyzer=\"word\",lowercase=False)\n",
    "analyzer=vectorizer.build_analyzer()\n",
    "analyzer(\"I, have a dog\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vectorizing as a sequence\n",
    "\n",
    "* Each document is a row\n",
    "* Words are turned into indices, their order is preserved\n",
    "* We will have to introduce padding, since documents are of different lengths, but we will need to have a array\n",
    "* Padding: fill shorter documents with zeros at end until the length of the longest document is reached"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[37, 2370], [21, 2370, 14, 154, 6, 1153, 388, 939], [915, 58, 94, 3512]]\n"
     ]
    }
   ],
   "source": [
    "def vectorize_into_sequences(texts,analyzer,vector_model):\n",
    "    result=[] #all docs, list of lists\n",
    "    for document in texts:\n",
    "        doc=[] #one doc\n",
    "        for w in analyzer(document): #tokenize\n",
    "            if w in vector_model.vocab: #is it in the vocab?\n",
    "                doc.append(vector_model.vocab[w].index+1) #+1 to make space for padding\n",
    "        result.append(doc)\n",
    "    return result\n",
    "\n",
    "seq=vectorize_into_sequential(texts,analyzer,vector_model)\n",
    "\n",
    "print(vectorize_into_sequential([\"I have a dog!\", \"The dog is used to produce a long sentence.\", \"Not so my cat.\"], analyzer, vector_model))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* above is the vectorized data before padding\n",
    "* padding is quite easy, in the end:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (25000, 2273)\n",
      "First example: [ 132 1115   23 ...    0    0    0]\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.sequence import pad_sequences\n",
    "vectorized_data_padded=pad_sequences(seq, padding='post')\n",
    "print(\"Shape:\", vectorized_data_padded.shape)\n",
    "print(\"First example:\", vectorized_data_padded[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and that is our data, nicely padded"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Labels into numerical vectors\n",
    "\n",
    "- Same as in the original BOW classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "class_numbers shape= (25000,)\n",
      "class_numbers [1 0 1 ... 0 0 0]\n",
      "class labels ['neg' 'pos']\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "label_encoder=LabelEncoder() #Turns class labels into integers\n",
    "class_numbers=label_encoder.fit_transform(labels)\n",
    "print(\"class_numbers shape=\",class_numbers.shape)\n",
    "print(\"class_numbers\",class_numbers)\n",
    "print(\"class labels\",label_encoder.classes_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Network\n",
    "\n",
    "* The embedding matrix can be obtained straight from the vector_model\n",
    "* We have a little problem, though because we added a padding symbol at index 0\n",
    "* So now we need to add a row of zeros for it, or else our embedding lookup will be off by one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Orig shape: (100000, 300) float32\n",
      "New  shape: (100001, 300)\n",
      "First two rows: [[ 0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00\n",
      "   0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00  0.0000e+00]\n",
      " [ 1.0730e-01  8.9000e-03  6.0000e-04  5.5000e-03 -6.4600e-02 -6.0000e-02\n",
      "   4.5000e-02 -1.3300e-02 -3.5700e-02  4.3000e-02 -3.5600e-02 -3.2000e-03\n",
      "   7.3000e-03 -1.0000e-04  2.5800e-02 -1.6600e-02  7.5000e-03  6.8600e-02\n",
      "   3.9200e-02  7.5300e-02  1.1500e-02 -8.7000e-03  4.2100e-02  2.6500e-02\n",
      "  -6.0100e-02  2.4200e-01  1.9900e-02 -7.3900e-02 -3.1000e-03 -2.6300e-02\n",
      "  -6.2000e-03  1.6800e-02 -3.5700e-02 -2.4900e-02  1.9000e-02 -1.8400e-02\n",
      "  -5.3700e-02  1.4200e-01  6.0000e-02  2.2600e-02 -3.8000e-03 -6.7500e-02\n",
      "  -3.6000e-03 -8.0000e-03  5.7000e-02  2.0800e-02  2.2300e-02 -2.5600e-02\n",
      "  -1.5300e-02  2.2000e-03 -4.8200e-02  1.3100e-02 -6.0160e-01 -8.8000e-03\n",
      "   1.0600e-02  2.2900e-02  3.3600e-02  7.1000e-03  8.8700e-02  2.3700e-02\n",
      "  -2.9000e-02 -4.0500e-02 -1.2500e-02  1.4700e-02  4.7500e-02  6.4700e-02\n",
      "   4.7400e-02  1.9900e-02  4.0800e-02  3.2200e-02  3.6000e-03  3.5000e-02\n",
      "  -7.2300e-02 -3.0500e-02  1.8400e-02 -2.6000e-03  2.4000e-02 -1.6000e-02\n",
      "  -3.0800e-02  4.3400e-02  1.4700e-02 -4.5700e-02 -2.6700e-02 -1.7030e-01\n",
      "  -9.9000e-03  4.1700e-02  2.3500e-02 -2.6000e-02 -1.5190e-01 -1.1600e-02\n",
      "  -3.0600e-02 -4.1300e-02  3.3000e-02  7.2300e-02  3.6500e-02 -1.0000e-04\n",
      "   4.2000e-03  3.4600e-02  2.7700e-02 -3.0500e-02  7.8400e-02 -4.0400e-02\n",
      "   1.8700e-02 -2.2500e-02 -2.0600e-02 -1.7900e-02 -2.4280e-01  6.6900e-02\n",
      "   5.2300e-02  5.2700e-02  1.4900e-02 -7.0800e-02 -9.8700e-02  2.6300e-02\n",
      "  -6.1100e-02  3.0200e-02  2.1600e-02  3.1300e-02 -1.4000e-02 -2.4950e-01\n",
      "  -3.4600e-02 -4.8000e-02  2.5000e-02  2.1300e-01 -3.3000e-02 -1.5530e-01\n",
      "  -2.9200e-02 -3.4600e-02  1.0740e-01  1.0000e-03 -1.1700e-02 -5.7000e-03\n",
      "  -1.2800e-01 -3.8000e-03  1.3000e-02 -1.1570e-01 -1.0800e-02  2.7500e-02\n",
      "   1.5800e-02 -1.6900e-02  7.0000e-03  2.4700e-02  5.1000e-02  1.0292e+00\n",
      "  -2.8300e-02 -3.1000e-02 -2.6000e-03 -3.4300e-02  5.7800e-02  4.4400e-02\n",
      "   8.1200e-02 -2.1100e-02 -8.7200e-02  1.6900e-02  4.9900e-02  4.8500e-02\n",
      "   2.2700e-02 -3.2300e-02 -3.5000e-03  4.3500e-02 -2.7500e-02  1.5400e-02\n",
      "   1.3500e-02 -4.8400e-02 -6.9900e-02 -5.0200e-02  2.7450e-01 -3.0000e-04\n",
      "  -3.7100e-02  5.1700e-02 -9.0800e-02  1.3000e-03  3.6000e-02  2.8000e-02\n",
      "   8.3900e-02  9.8000e-02 -4.9000e-02 -2.4230e-01 -1.4200e-02  2.4000e-03\n",
      "  -2.0700e-02  1.2000e-03  8.8000e-03 -1.4300e-02 -1.9700e-02  5.1500e-02\n",
      "  -8.5000e-03  2.5700e-02  2.1540e-01  3.0100e-02  2.1100e-02  5.3000e-02\n",
      "  -5.0000e-04  1.7700e-02  1.6000e-03 -5.3000e-03 -1.6200e-02 -2.2300e-02\n",
      "  -1.8620e-01  3.9800e-02  6.5800e-02 -9.6200e-02 -7.6000e-03 -7.5000e-03\n",
      "  -3.4200e-02 -2.6500e-02  4.2000e-02  5.2200e-02 -2.6600e-02  2.0100e-02\n",
      "  -1.3310e-01 -3.6700e-02  3.5100e-02  5.1800e-02 -8.7000e-03  5.9900e-02\n",
      "  -1.0860e-01 -1.8800e-02  4.8100e-02  1.0500e-02 -6.0000e-03  1.5100e-02\n",
      "  -3.1000e-03  7.7000e-03 -2.7600e-02 -3.7300e-02 -2.0300e-02  4.7200e-02\n",
      "   2.4600e-02  1.4400e-01  5.4200e-02 -2.2500e-02  2.4950e-01  1.6170e-01\n",
      "   3.8000e-03  1.1190e-01 -2.3000e-02 -7.8500e-02  2.5000e-02 -6.1600e-02\n",
      "  -4.8500e-02  2.2500e-02  2.8100e-02  4.1000e-03  1.1200e-02  1.7200e-02\n",
      "   2.9100e-02 -2.8200e-02  2.6000e-03  4.0550e-01  3.9200e-02  8.8000e-03\n",
      "   2.2800e-02  2.9900e-02  1.1950e-01  5.4500e-02 -2.0000e-03  2.0000e-03\n",
      "   4.9000e-02  1.4500e-02 -8.6000e-03  9.8000e-03 -2.3600e-02  1.7100e-02\n",
      "  -7.6500e-02 -4.0000e-02  1.2800e-02  1.1000e-03  4.2000e-03  2.4400e-02\n",
      "   7.5000e-03  2.0000e-02  2.0100e-02  1.9600e-02 -3.7700e-02 -4.3200e-02\n",
      "  -7.3000e-03 -2.1000e-03  1.8300e-02  7.6000e-03  1.8050e-01 -5.5100e-02\n",
      "   7.5000e-03 -5.1600e-02  4.2000e-02 -6.8000e-03 -7.1100e-02 -1.4080e-01\n",
      "   5.0400e-02  2.7600e-02  4.7000e-02  3.2300e-02 -2.1900e-02  1.0000e-03\n",
      "   8.9000e-03  2.7600e-02  1.8600e-02  5.0000e-03  1.1730e-01 -4.0000e-02]]\n"
     ]
    }
   ],
   "source": [
    "# This is where the embedding matrix is\n",
    "orig_embedding_matrix=vector_model.vectors\n",
    "print(\"Orig shape:\",orig_embedding_matrix.shape, orig_embedding_matrix.dtype)\n",
    "zero_line=numpy.zeros((1,orig_embedding_matrix.shape[1]),dtype=orig_embedding_matrix.dtype)\n",
    "#Stack the zeros on top of the embedding matrix\n",
    "embedding_matrix=numpy.vstack((zero_line,orig_embedding_matrix))\n",
    "print(\"New  shape:\",embedding_matrix.shape)\n",
    "print(\"First two rows:\", embedding_matrix[:2,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "source": [
    "### Sequential input\n",
    "\n",
    "- Remember how the shape of the input data matrix had undefined number of columns\n",
    "- Now we must make it into fixed size (same for each example)\n",
    "- Padding: include zeros until you reach the correct size\n",
    "- You will hear more about this next week!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Our network structure:\n",
    "\n",
    "- Input layer, Embedding layer with pretrained weights, Average of embeddings, Non-linear activation, Classification layer\n",
    "- The key point here is the embedding layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_3\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_5 (InputLayer)         (None, 2273)              0         \n",
      "_________________________________________________________________\n",
      "embedding_5 (Embedding)      (None, 2273, 300)         30000300  \n",
      "_________________________________________________________________\n",
      "global_average_pooling1d_2 ( (None, 300)               0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 50)                15050     \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 2)                 102       \n",
      "=================================================================\n",
      "Total params: 30,015,452\n",
      "Trainable params: 30,015,452\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Train on 22500 samples, validate on 2500 samples\n",
      "Epoch 1/50\n",
      "22500/22500 [==============================] - 221s 10ms/step - loss: 0.4113 - accuracy: 0.8184 - val_loss: 0.2762 - val_accuracy: 0.8816\n",
      "Epoch 2/50\n",
      "14100/22500 [=================>............] - ETA: 1:18 - loss: 0.1802 - accuracy: 0.9348"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-24-26f0311716d3>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     23\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     24\u001b[0m \u001b[0;31m# train\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 25\u001b[0;31m \u001b[0mhist\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvectorized_data_padded\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mclass_numbers\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mverbose\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m50\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mvalidation_split\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/venv-jlab/lib/python3.7/site-packages/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)\u001b[0m\n\u001b[1;32m   1237\u001b[0m                                         \u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1238\u001b[0m                                         \u001b[0mvalidation_steps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mvalidation_steps\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1239\u001b[0;31m                                         validation_freq=validation_freq)\n\u001b[0m\u001b[1;32m   1240\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1241\u001b[0m     def evaluate(self,\n",
      "\u001b[0;32m~/venv-jlab/lib/python3.7/site-packages/keras/engine/training_arrays.py\u001b[0m in \u001b[0;36mfit_loop\u001b[0;34m(model, fit_function, fit_inputs, out_labels, batch_size, epochs, verbose, callbacks, val_function, val_inputs, shuffle, initial_epoch, steps_per_epoch, validation_steps, validation_freq)\u001b[0m\n\u001b[1;32m    194\u001b[0m                     \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtoarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    195\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 196\u001b[0;31m                 \u001b[0mouts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfit_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mins_batch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    197\u001b[0m                 \u001b[0mouts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mto_list\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mouts\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    198\u001b[0m                 \u001b[0;32mfor\u001b[0m \u001b[0ml\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mo\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mout_labels\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mouts\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/venv-jlab/lib/python3.7/site-packages/tensorflow/python/keras/backend.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m   3290\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3291\u001b[0m     fetched = self._callable_fn(*array_vals,\n\u001b[0;32m-> 3292\u001b[0;31m                                 run_metadata=self.run_metadata)\n\u001b[0m\u001b[1;32m   3293\u001b[0m     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call_fetch_callbacks\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfetched\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3294\u001b[0m     output_structure = nest.pack_sequence_as(\n",
      "\u001b[0;32m~/venv-jlab/lib/python3.7/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1456\u001b[0m         ret = tf_session.TF_SessionRunCallable(self._session._session,\n\u001b[1;32m   1457\u001b[0m                                                \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_handle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1458\u001b[0;31m                                                run_metadata_ptr)\n\u001b[0m\u001b[1;32m   1459\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1460\u001b[0m           \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import Input, Dense, Embedding, Activation, GlobalAveragePooling1D\n",
    "from keras.optimizers import SGD, Adam\n",
    "from keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "\n",
    "\n",
    "example_count,sequence_len=vectorized_data_padded.shape\n",
    "class_count=len(label_encoder.classes_)\n",
    "vector_size=embedding_matrix.shape[1] # embedding dim (\"hidden layer\") must be the same as in the pretrained model\n",
    "vocab_size=embedding_matrix.shape[0]\n",
    "\n",
    "inp=Input(shape=(sequence_len,))\n",
    "embeddings=Embedding(vocab_size, vector_size, mask_zero=True, weights=[embedding_matrix])(inp)\n",
    "average_embeddings=GlobalAveragePooling1D()(embeddings) # is masking-aware\n",
    "hidden=Dense(50,activation=\"tanh\")(average_embeddings)\n",
    "outp=Dense(class_count, activation=\"softmax\")(hidden)\n",
    "model=Model(inputs=[inp], outputs=[outp])\n",
    "\n",
    "optimizer=Adam(lr=0.001) # define the learning rate\n",
    "model.compile(optimizer=optimizer,loss=\"sparse_categorical_crossentropy\",metrics=['accuracy'])\n",
    "\n",
    "print(model.summary())\n",
    "\n",
    "# train\n",
    "stop_cb=EarlyStopping(monitor='val_loss', patience=2, verbose=1, mode='auto', baseline=None, restore_best_weights=True)\n",
    "hist=model.fit(vectorized_data_padded,class_numbers,batch_size=100,verbose=1,epochs=50,validation_split=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "print(\"History:\",hist.history[\"val_acc\"])\n",
    "print(\"Max accuracy:\",numpy.max(hist.history[\"val_acc\"]))\n",
    "plt.ylim(0.85,1.0)\n",
    "plt.plot(hist.history[\"val_acc\"],label=\"Validation set accuracy\")\n",
    "plt.plot(hist.history[\"acc\"],label=\"Training set accuracy\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}