lecture_chap09_exercise_master.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# 第9回講義 演習"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from sklearn.utils import shuffle\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from keras.datasets import imdb\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "rng = np.random.RandomState(1234)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 課題1. グラフ上でのLoop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "tensorflowの計算グラフ上でloop構造を実現するには, `tf.scan`関数を使用します"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### tf.scan関数\n",
    "- 主な引数\n",
    "    - fn: 入力系列に適用する関数\n",
    "    - elems: 入力系列 (第0軸方向に走査していく)\n",
    "    - initializer: 最初の引数\n",
    "    \n",
    "参考:\n",
    "https://www.tensorflow.org/api_docs/python/tf/scan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### tf.scanの機能と注意事項\n",
    "\n",
    "まず, 入力系列に対して適用する関数fnは, fn(a, x)といった様に, 2つの引数を持つものである必要があります.\n",
    "\n",
    "この2つの引数にはそれぞれ役割があり, 次のようになっています.\n",
    "  - 第1引数: 前ステップのfnの出力\n",
    "  - 第2引数: 今ステップの入力(elems)\n",
    "  \n",
    "つまり, 出力される系列は, 例えばelemsの長さがNであれば,\n",
    "\n",
    "$f_1={\\rm fn}(initializer, elems[0])$\n",
    "\n",
    "$f_2={\\rm fn}(f_1, elems[1])$\n",
    "\n",
    "$f_3={\\rm fn}(f_2, elems[2])$\n",
    "\n",
    "$\\vdots$\n",
    "\n",
    "$f_N={\\rm fn}(f_{N-1}, elems[N-1])$\n",
    "\n",
    "ということになります."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### 例:Accumulation function for vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32)\n",
    "\n",
    "def fn(a, x):\n",
    "    return a + x\n",
    "\n",
    "res = tf.scan(fn=fn, elems=x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    print(sess.run(res, feed_dict={x: np.array([1, 2, 3, 4, 5, 6])}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### 例:Accumulation function for matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32)\n",
    "\n",
    "def fn(a, x):\n",
    "    return a + x\n",
    "\n",
    "res = tf.scan(fn=fn, elems=x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    print(sess.run(res, feed_dict={\n",
    "            x: np.array([[1, 2, 3, 4, 5],\n",
    "                         [1, 2, 3, 4, 5],\n",
    "                         [1, 2, 3, 4, 5]])\n",
    "    }))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### 例: initializer\n",
    "* tf.scanのinitializerという引数で，loop構造の初期値を明示的に指定します．特にinitializerが指定されない場合は，上記のように入力系列の最初が初期値となります．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "x = (tf.placeholder(tf.float32), tf.placeholder(tf.float32))\n",
    "init = tf.placeholder(tf.float32)\n",
    "\n",
    "def fn(a, x):\n",
    "    return x[0] - x[1] + a\n",
    "\n",
    "res = tf.scan(fn=fn, elems=x, initializer=init)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "elems = np.array([1, 2, 3, 4, 5, 6])\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(res, feed_dict={\n",
    "            x: (elems+1, elems),\n",
    "            init: np.array(0)\n",
    "    }))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### 例: フィボナッチ数列（initializerを利用）\n",
    "$F_0 = 0,$\n",
    "$F_1 = 1,$\n",
    "$F_{n + 2} = F_n + F_{n + 1} (n ≧ 0)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32)\n",
    "init = (tf.placeholder(tf.float32), tf.placeholder(tf.float32))\n",
    "\n",
    "def fn(a, _):\n",
    "    return (a[1], a[0]+a[1])\n",
    "\n",
    "res = tf.scan(fn=fn, elems=x, initializer=init)\n",
    "\n",
    "# fibonaccis == ([1, 1, 2, 3, 5, 8], [1, 2, 3, 5, 8, 13])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    print(sess.run(res, feed_dict={\n",
    "            x: np.array([0, 0, 0, 0, 0, 0]),\n",
    "            init: (np.array(0), np.array(1))\n",
    "    }))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 課題2. Recurrent Neural Network (RNN) によるIMDbのsentiment analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "IMDb (Internet Movie Database) と呼ばれる映画レビューのデータセットで\n",
    "\n",
    "各レビュー文の評価がpositiveかnegativeかをRNNを用いて予測してみましょう.\n",
    "\n",
    "<div style=\"text-align: center;\">【データセットのイメージ】</div>\n",
    "\n",
    "| レビュー | 評価 |\n",
    "|:--------:|:-------------:|\n",
    "|Where's Michael Caine when you need him? I've ...|negative|\n",
    "|To experience Head you really need to understa...|positive|\n",
    "\n",
    "※実際には各単語が出現頻度順位で数字に置き換えられたものがXとして, 評価をnegativeなら0, positiveなら1に置き換えたものがyとして入ることになります."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 1. データセットの読み込み"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# 出現頻度上位num_words番までのみを扱う. それ以外は丸ごと1つの定数に置き換え(0:start_char, 1:oov_char, 2~:word_index)\n",
    "num_words = 10000\n",
    "(train_X, train_y), (test_X, test_y) = imdb.load_data(num_words=num_words, start_char=0, oov_char=1, index_from=2)\n",
    "\n",
    "# split data into training and validation\n",
    "train_X, valid_X, train_y, valid_y = train_test_split(train_X, train_y, test_size=0.2, random_state=42)\n",
    "\n",
    "# データセットサイズが大きいので演習用に短縮\n",
    "train_X = train_X[:len(train_X)//2]\n",
    "train_y = train_y[:len(train_y)//2]\n",
    "valid_X = valid_X[:len(valid_X)//2]\n",
    "valid_y = valid_y[:len(valid_y)//2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 2. 可変長系列のミニバッチ化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "IMDbの各データは長さの異なるレビュー (の各単語を出現頻度順で数値化したもの) です. これに対してRNNを適用し, 最後の隠れ層ベクトルを元に二値分類をおこないます.\n",
    "\n",
    "この問題で異なる長さの系列をミニバッチ化する際には次の2つのことに注意する必要があります.\n",
    "\n",
    "- ミニバッチ内のデータの系列の長さをpaddingによって揃える.\n",
    "- paddingした部分の計算を無効にする"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### 2.1. ミニバッチ内のデータの系列の長さをpaddingによって揃える."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "異なる系列長のデータをミニバッチ(行列)に落とし込むために, ミニバッチ内の短い系列に対して頭orお尻にpaddingし長さを揃える必要があります. これは `keras` にある関数 `pad_sequences` を使うなどすればできます. またpaddingの量を少なくするために, あらかじめデータの長さで降順にソートしておくことが多いです."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### 2.2. paddingした部分の計算を無効にする"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "paddingの部分はあくまで系列長を合わせるためなので, 通常のRNNの計算はおこなわず, 何らかの形で計算を無効にする必要があります. ここではわかりやすい実装として, paddingの部分では代わりに前のステップの隠れ層をコピーするようにし, 実際の系列の最後の単語における隠れ層ベクトルを保持するようにします."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "具体的には, 各インスタンスに対して実際に単語がある部分に1, ない部分(paddingの部分)に0を置くバイナリのマスク$m=[m_1, m_2, \\dots, m_t, \\dots, m_T]$をつくり,\n",
    "\n",
    "$$\n",
    "    h_t = m_t \\cdot \\sigma({\\bf W_x} x_t + {\\bf W_h} h_{t-1} + b) + (1-m_t) \\cdot h_{t-1}\n",
    "$$\n",
    "\n",
    "とします. こうすることでpaddingの部分では$h_t=h_{t-1}$となり, paddingの計算結果に対する影響がなくなります."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3. 各層クラスの実装"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### 3.1. Embedding層\n",
    "\n",
    "Embedding層では, 単語を離散的なidから連続的な数百次元のベクトルに変換(埋め込み, embed)します.\n",
    "\n",
    "下のEmbeddingクラスにおいて, 入力`x`は各行に文の単語のid列が入った行列で, 重み`V`は各行がそれぞれの単語idのベクトルに対応した行列です. つまりそれぞれの行列のサイズは\n",
    "\n",
    "- `x`: (ミニバッチサイズ) x (ミニバッチ内の文の最大系列長)\n",
    "- `V`: (辞書の単語数) x (単語のベクトルの次元数)\n",
    "\n",
    "です.\n",
    "\n",
    "この`V`から, 入力`x`のそれぞれの単語idに対して対応する単語ベクトルを取り出すことで, 各単語をベクトルに変換します. \n",
    "\n",
    "`tf`では`tf.nn.embedding_lookup`によりこの作業を行います.この処理によって出力されるテンソルの次元数は，(ミニバッチサイズ) x (ミニバッチ内の文の最大系列長) x (単語のベクトルの次元数)となります（embedding層に関する詳細は，次の第10回講義で説明があります）．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "class Embedding:\n",
    "    def __init__(self, vocab_size, emb_dim, scale=0.08):\n",
    "        self.V = tf.Variable(rng.randn(vocab_size, emb_dim).astype('float32') * scale, name='V')\n",
    "\n",
    "    def f_prop(self, x):\n",
    "        return tf.nn.embedding_lookup(self.V, x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2. RNN\n",
    "\n",
    "RNNクラスでは, Embedding層で各単語がベクトルに変換されたものを入力として処理を行います. ここで入力`x`は\n",
    "\n",
    "- `x`: (ミニバッチサイズ) x (ミニバッチ内の文の最大系列長) x (単語のベクトルの次元数)\n",
    "\n",
    "となっています. `tf.scan`では第0軸方向に走査していくので, 文の系列方向に沿って走査するために上の第0軸と第1軸を入れ替えて\n",
    "\n",
    "- `x`: (ミニバッチ内の文の最大系列長) x (ミニバッチサイズ) x (単語のベクトルの次元数)\n",
    "\n",
    "とします."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#  Random orthogonal initializer (see [Saxe et al. 2013])\n",
    "def orthogonal_initializer(shape, scale = 1.0):\n",
    "    a = np.random.normal(0.0, 1.0, shape).astype(np.float32)\n",
    "    u, _, v = np.linalg.svd(a, full_matrices=False)\n",
    "    q = u if u.shape == shape else v\n",
    "    return scale * q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "class RNN:\n",
    "    def __init__(self, in_dim, hid_dim, m, scale=0.08):\n",
    "        self.in_dim = in_dim\n",
    "        self.hid_dim = hid_dim\n",
    "        # Xavier initializer\n",
    "        self.W_in = tf.Variable(rng.uniform(\n",
    "                        low=-np.sqrt(6/(in_dim + hid_dim)),\n",
    "                        high=np.sqrt(6/(in_dim + hid_dim)),\n",
    "                        size=(in_dim, hid_dim)\n",
    "                    ).astype('float32'), name='W_in')\n",
    "        # Random orthogonal initializer\n",
    "        self.W_re = tf.Variable(orthogonal_initializer((hid_dim, hid_dim)), name='W_re')\n",
    "        self.b_re = tf.Variable(tf.zeros([hid_dim], dtype=tf.float32), name='b_re')\n",
    "        self.m = m\n",
    "\n",
    "    def f_prop(self, x):\n",
    "        def fn(h_tm1, x_and_m):\n",
    "            x = x_and_m[0]\n",
    "            m = x_and_m[1]\n",
    "            h_t = tf.nn.tanh(tf.matmul(h_tm1, self.W_re) + tf.matmul(x, self.W_in) + self.b_re)\n",
    "            return m[:, np.newaxis] * h_t + (1 - m[:, np.newaxis]) * h_tm1 # Mask\n",
    "\n",
    "        # shape: [batch_size, sentence_length, in_dim] -> shape: [sentence_length, batch_size, in_dim]\n",
    "        _x = tf.transpose(x, perm=[1, 0, 2])\n",
    "        # shape: [batch_size, sentence_length] -> shape: [sentence_length, batch_size]\n",
    "        _m = tf.transpose(self.m)\n",
    "        h_0 = tf.matmul(x[:, 0, :], tf.zeros([self.in_dim, self.hid_dim])) # Initial state\n",
    "        \n",
    "        h = tf.scan(fn=fn, elems=[_x, _m], initializer=h_0)\n",
    "        \n",
    "        return h[-1] # Take the last state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "class Dense:\n",
    "    def __init__(self, in_dim, out_dim, function=lambda x: x):\n",
    "        # Xavier initializer\n",
    "        self.W = tf.Variable(rng.uniform(\n",
    "                        low=-np.sqrt(6/(in_dim + out_dim)),\n",
    "                        high=np.sqrt(6/(in_dim + out_dim)),\n",
    "                        size=(in_dim, out_dim)\n",
    "                    ).astype('float32'), name='W')\n",
    "        self.b = tf.Variable(np.zeros([out_dim]).astype('float32'))\n",
    "        self.function = function\n",
    "\n",
    "    def f_prop(self, x):\n",
    "        return self.function(tf.matmul(x, self.W) + self.b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 4. 計算グラフ構築 & パラメータの更新設定"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "emb_dim = 100\n",
    "hid_dim = 50\n",
    "\n",
    "x = tf.placeholder(tf.int32, [None, None], name='x')\n",
    "m = tf.cast(tf.not_equal(x, -1), tf.float32) # Mask. Paddingの部分(-1)は0, 他の値は1\n",
    "t = tf.placeholder(tf.float32, [None, None], name='t')\n",
    "\n",
    "layers = [\n",
    "    Embedding(num_words, emb_dim),\n",
    "    RNN(emb_dim, hid_dim, m=m),\n",
    "    Dense(hid_dim, 1, tf.nn.sigmoid)\n",
    "]\n",
    "\n",
    "def f_props(layers, x):\n",
    "    for i, layer in enumerate(layers):\n",
    "        x = layer.f_prop(x)\n",
    "    return x\n",
    "\n",
    "y = f_props(layers, x)\n",
    "\n",
    "cost = tf.reduce_mean(-t*tf.log(tf.clip_by_value(y, 1e-10, 1.0)) - (1. - t)*tf.log(tf.clip_by_value(1.-y, 1e-10, 1.0)))\n",
    "\n",
    "train = tf.train.AdamOptimizer().minimize(cost)\n",
    "test = tf.round(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 5. 学習"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Sort train data according to its length\n",
    "train_X_lens = [len(com) for com in train_X]\n",
    "sorted_train_indexes = sorted(range(len(train_X_lens)), key=lambda x: -train_X_lens[x])\n",
    "\n",
    "train_X = [train_X[ind] for ind in sorted_train_indexes]\n",
    "train_y = [train_y[ind] for ind in sorted_train_indexes]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "n_epochs = 5\n",
    "batch_size = 100\n",
    "n_batches_train = len(train_X) // batch_size\n",
    "n_batches_valid = len(valid_X) // batch_size\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init = tf.global_variables_initializer()\n",
    "    sess.run(init)\n",
    "    for epoch in range(n_epochs):\n",
    "        # Train\n",
    "        train_costs = []\n",
    "        for i in range(n_batches_train):\n",
    "            start = i * batch_size\n",
    "            end = start + batch_size\n",
    "            \n",
    "            train_X_mb = np.array(pad_sequences(train_X[start:end], padding='post', value=-1)) # Padding\n",
    "            train_y_mb = np.array(train_y[start:end])[:, np.newaxis]\n",
    "\n",
    "            _, train_cost = sess.run([train, cost], feed_dict={x: train_X_mb, t: train_y_mb})\n",
    "            train_costs.append(train_cost)\n",
    "        \n",
    "        # Valid\n",
    "        valid_costs = []\n",
    "        pred_y = []\n",
    "        for i in range(n_batches_valid):\n",
    "            start = i * batch_size\n",
    "            end = start + batch_size\n",
    "            \n",
    "            valid_X_mb = np.array(pad_sequences(valid_X[start:end], padding='post', value=-1)) # Padding\n",
    "            valid_y_mb = np.array(valid_y[start:end])[:, np.newaxis]\n",
    "            \n",
    "            pred, valid_cost = sess.run([test, cost], feed_dict={x: valid_X_mb, t: valid_y_mb})\n",
    "            pred_y += pred.flatten().tolist()\n",
    "            valid_costs.append(valid_cost)\n",
    "        print('EPOCH: %i, Training cost: %.3f, Validation cost: %.3f, Validation F1: %.3f' % (epoch+1, np.mean(train_costs), np.mean(valid_costs), f1_score(valid_y, pred_y, average='macro')))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}