text_classfiy_imdb.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 文本分类——以IMDB为例\n",
    "\n",
    "另一个Hello world级别的例子，学习基本的分类。\n",
    "\n",
    "IMDB数据集是对不同电影的评论。对于每一条评论都有一个标签，0代表负面,1代表正面。\n",
    "\n",
    "所以IMDB数据集用来训练分类模型是一个二分类问题。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 获取数据集\n",
    "\n",
    "使用keras获取数据集：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training entries: 25000, labels: 25000\n"
     ]
    }
   ],
   "source": [
    "from tensorflow import keras\n",
    "\n",
    "imdb = keras.datasets.imdb\n",
    "# 首次运行，会从网络下载imdb.npz文件到用户目录，以我为例/home/allen/.keras/datasets/imdb.npz\n",
    "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)\n",
    "print(\"Training entries: {}, labels: {}\".format(len(train_data), len(train_labels)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面获取的数据都是整形值，每一个值代表着一个词语。\n",
    "\n",
    "因此，我们需要通过id将字符串取出来。\n",
    "\n",
    "做法如下：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training entries: 25000, labels: 25000\n",
      "<PAD>,<START>,<UNK>,<UNUSED>\n"
     ]
    }
   ],
   "source": [
    "from tensorflow import keras\n",
    "\n",
    "# 构造出字符到id的映射\n",
    "# 首次运行会从网络下载imdb_word_index.json文件到用户目录，以我为例/home/allen/.keras/datasets/imdb_word_index.json\n",
    "word2idx = imdb.get_word_index()\n",
    "\n",
    "# 将word2idx中每一个词的索引值，往后推移3，因为我们需要在开始插入4个字符。\n",
    "# 要插入4个字符，却只推移3，是因为原来的单词与id之间的对于关系中,id是从1开始的,\n",
    "# 并且这个1代表的是一条评论开始，每个train_data都是以1开始！！\n",
    "word2idx = {k: (v + 3) for k, v in word2idx.items()}\n",
    "# 用来对齐，在短句子后面补齐该字符\n",
    "word2idx['<PAD>'] = 0\n",
    "# 用来标记句子的起始位置\n",
    "word2idx['<START>'] = 1\n",
    "# 用来标记未知词，因为word2idx只有10000个词语，有可能不完整\n",
    "word2idx['<UNK>'] = 2\n",
    "# 用来标记未使用词\n",
    "word2idx['<UNUSED>'] = 3\n",
    "\n",
    "# 构造出id到字符的映射\n",
    "idx2word = dict([(value, key) for key, value in word2idx.items()])\n",
    "print(\"%s,%s,%s,%s\" % (\n",
    "  idx2word.get(0, \"?\"), idx2word.get(1, \"?\"), idx2word.get(2, \"?\"),\n",
    "  idx2word.get(3, \"?\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，对于每一个句子的id表示，我们需要将它转化为字符串。\n",
    "\n",
    "我们定义一个函数来完成这个转换：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decode_review(text_ids):\n",
    "    return \" \".join([id2word.get(i, \"?\") for i in text_ids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 预处理数据\n",
    "\n",
    "因为电影评论的句子长度不一，我们需要将它对齐：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 对训练数据进行对齐，方法为在每个句子末尾追加`<PAD>`字符对齐，最大长度限制为256个词语\n",
    "train_data = keras.preprocessing.sequence.pad_sequences(\n",
    "  train_data,# 是ID值\n",
    "  value=word2idx[\"<PAD>\"],\n",
    "  padding='post',\n",
    "  maxlen=256)\n",
    "print(len(train_data[0]), len(train_data[1]))\n",
    "print(train_data[0])\n",
    "print(decode_review(train_data[0]))\n",
    "\n",
    "test_data = keras.preprocessing.sequence.pad_sequences(\n",
    "  test_data,\n",
    "  value=word2idx[\"<PAD>\"],\n",
    "  padding='post',\n",
    "  maxlen=256)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "输出结果如下：\n",
    "\n",
    "```bash\n",
    "256 256\n",
    "[   1   14   22   16   43  530  973 1622 1385   65  458 4468   66 3941\n",
    "    4  173   36  256    5   25  100   43  838  112   50  670    2    9\n",
    "   35  480  284    5  150    4  172  112  167    2  336  385   39    4\n",
    "  172 4536 1111   17  546   38   13  447    4  192   50   16    6  147\n",
    " 2025   19   14   22    4 1920 4613  469    4   22   71   87   12   16\n",
    "   43  530   38   76   15   13 1247    4   22   17  515   17   12   16\n",
    "  626   18    2    5   62  386   12    8  316    8  106    5    4 2223\n",
    " 5244   16  480   66 3785   33    4  130   12   16   38  619    5   25\n",
    "  124   51   36  135   48   25 1415   33    6   22   12  215   28   77\n",
    "   52    5   14  407   16   82    2    8    4  107  117 5952   15  256\n",
    "    4    2    7 3766    5  723   36   71   43  530  476   26  400  317\n",
    "   46    7    4    2 1029   13  104   88    4  381   15  297   98   32\n",
    " 2071   56   26  141    6  194 7486   18    4  226   22   21  134  476\n",
    "   26  480    5  144   30 5535   18   51   36   28  224   92   25  104\n",
    "    4  226   65   16   38 1334   88   12   16  283    5   16 4472  113\n",
    "  103   32   15   16 5345   19  178   32    0    0    0    0    0    0\n",
    "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
    "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
    "    0    0    0    0]\n",
    "    \n",
    "<START> this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert <UNK> is an amazing actor and now the same being director <UNK> father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for <UNK> and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also <UNK> to the two little boy's that played the <UNK> of norman and paul they were just brilliant children are often left out of the <UNK> list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n",
    "```\n",
    "可以发现，句子末尾全部是数字0，因为`<PAD>`的id值是0。\n",
    "因为train_data每个句子都是以1开始，所以转化成字符串之后，都会有`<START>`这个字符。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建模型\n",
    "\n",
    "还是使用keras的序贯模型：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras\n",
    "\n",
    "vocab_size = 10000\n",
    "\n",
    "model = keras.Sequential()\n",
    "# 词嵌入，每个词语用16个浮点数组成的一维向量表示\n",
    "model.add(keras.layers.Embedding(vocab_size, 16))\n",
    "# 平均池化，下面再解释\n",
    "model.add(keras.layers.GlobalAveragePooling1D())\n",
    "# 全连接层，使用ReLU激活函数\n",
    "model.add(keras.layers.Dense(16, activation=keras.activations.relu))\n",
    "# 输出层，使用sigmoid函数，将输出数值压扁在(0,1)区间\n",
    "model.add(keras.layers.Dense(1, activation=keras.activations.sigmoid))\n",
    "\n",
    "# 输出模型的综述\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "_________________________________________________________________\n",
    "Layer (type)                 Output Shape              Param #   \n",
    "=================================================================\n",
    "embedding_1 (Embedding)      (None, None, 16)          160000    \n",
    "_________________________________________________________________\n",
    "global_average_pooling1d_1 ( (None, 16)                0         \n",
    "_________________________________________________________________\n",
    "dense_1 (Dense)              (None, 16)                272       \n",
    "_________________________________________________________________\n",
    "dense_2 (Dense)              (None, 1)                 17        \n",
    "=================================================================\n",
    "Total params: 160,289\n",
    "Trainable params: 160,289\n",
    "Non-trainable params: 0\n",
    "_________________________________________________________________\n",
    "\n",
    "```\n",
    "\n",
    "`embedding_1`的output shape意思是(BATCH_SIZE, STEPS, FEATURES)，None代表任意值。\n",
    "`global_average_pooling1d_1`的output shape意思是(BATCH_SIZE, FEATURES)。\n",
    "`dense_1`的output shape意思也是(BATCH_SIZE, FEATURES)。\n",
    "\n",
    "平均池化什么意思呢？做什么用处呢？\n",
    "\n",
    "根据tensorflow的文档，这里的平均池化是在TIME_STEPS(STEPS)这个维度上据平均值，目的是处理变长的输入序列（STEPS代表序列的长度），将输入序列转化成一个固定长度的向量。使用平均池化可能是最简单的达成这个目的的做法。\n",
    "\n",
    "这里的平均池化实现也非常简单，就是在STEPS这个维度取平均值。\n",
    "以TensorFlow作为后端为例：\n",
    "\n",
    "```python\n",
    "import tensorflow as tf\n",
    "\n",
    "# inputs就是我们的输入向量，也就是Embedding层的输出，形状是(BATCH_SIZE, STEPS, FEATURES)\n",
    "# keepdim为False说明去掉在axis指定的维度，所以经过平均池化后，输出向量的形状变为(BATCH_SIZE, FEATURES)\n",
    "tf.mean(inputs, axis=1, keepdim=False)\n",
    "```\n",
    "\n",
    "最后的输出层，只有一个节点。因为我们的标签要么是0要么是1，实际上这就是一个二分类的问题。所以我们使用sigmoid函数，它将输出值压缩在0~1的范围。\n",
    "\n",
    "**为什么二分类问题就要用sigmoid函数呢**，请看知乎的回答：https://www.zhihu.com/question/35322351"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 编译模型\n",
    "\n",
    "因为是二分类，所以loss我们选择binary_crossentropy。\n",
    "我们使用Adam优化器。\n",
    "\n",
    "代码如下：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=keras.optimizers.Adam(),\n",
    "              loss=keras.losses.binary_crossentropy,\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练模型\n",
    "\n",
    "首先需要准备数据，将train_data分成两部分，一部分用来训练，一部分用来验证。并且我们这次训练使用mini-batch为512来训练模型。\n",
    "\n",
    "代码如下：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 前10000个数据用来验证，之后的用来训练\n",
    "x_val = train_data[:10000]\n",
    "partial_x_train = train_data[10000:]\n",
    "\n",
    "y_val = train_labels[:10000]\n",
    "partial_y_train = train_labels[10000:]\n",
    "\n",
    "# 训练40轮，每一轮使用validation_data进行验证\n",
    "history = model.fit(partial_x_train,\n",
    "                    partial_y_train,\n",
    "                    epochs=40,\n",
    "                    batch_size=512,\n",
    "                    validation_data=(x_val, y_val),\n",
    "                    verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 估算结果\n",
    "\n",
    "我们使用测试集进行估算：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = model.evaluate(test_data, test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "[0.34129497660636904, 0.86908]\n",
    "```\n",
    "结果是：loss值为0.341，正确率为0.869"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练过程数据的可视化\n",
    "\n",
    "为了更清楚地获取训练过程中的loss，val_loss，acc和val_acc的变化，我们进行可视化。\n",
    "\n",
    "model.fit()方法返回一个history对象，我们可以通过history对象获取这些信息。\n",
    "\n",
    "代码如下：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history_dict = history.history\n",
    "history_dict.keys()\n",
    "\n",
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "# \"bo\" is for \"blue dot\"\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "# b is for \"solid blue line\"\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()\n",
    "\n",
    "plt.clf()  # clear figure\n",
    "acc_values = history_dict['acc']\n",
    "val_acc_values = history_dict['val_acc']\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最终的结果如下图：\n",
    "\n",
    "![training_and_validation_loss](images/imdb_training_and_validation_loss.png)\n",
    "![training_and_validation_accuracy](images/imdb_training_and_validation_accuracy.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}