natural-language-processing/Skip-Thought-Vectors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Skip-Thought Vectors\n",
    "    Ryan Kiros,Yukun Zhu,Ruslan Salakhutdinov,Richard S. Zemel,Antonio Torralba, Raquel Urtasun, Sanja Fidler\n",
    "    2015\n",
    "\n",
    "http://papers.nips.cc/paper/5950-skip-thought-vectors.pdf\n",
    "\n",
    "https://github.com/ryankiros/skip-thoughts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "一种unsupervised的句子级别encoder方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Introduction\n",
    "An approach for unsupervised learning of a generic, distributed sentence encoder.\n",
    "\n",
    "![1](http://ou8qjsj0m.bkt.clouddn.com//17-8-6/31481124.jpg)\n",
    "\n",
    "One difficulty that arises with such an experimental setup is being able to construct a large enough word vocabulary to encode arbitrary sentences. For example, a sentence from a Wikipedia article might contain nouns that are highly unlikely to appear in our book vocabulary. We solve this problem by learning a mapping that transfers word representations from one model to another. Using pre-trained word2vec representations learned with a continuous bag-of-words model, we learn a linear mapping from a word in word2vec space to a word in the encoder’s vocabulary space. The mapping is learned using all words that are shared between vocabularies. After training, any word that appears in word2vec can then get a vector in the encoder word embedding space.\n",
    "\n",
    "##  Approach\n",
    "###  Inducing skip-thought vectors\n",
    "Encoder-decoder models:\n",
    "- encoder: maps words to a sentence vector\n",
    "- decoder: generate the surrounding sentences\n",
    "\n",
    "Define:\n",
    "- given a sentence tuple $(s_{i-1}, s_i, s_{i+1})$\n",
    "- $w_i^t$ : the t-th word for sentence $s_i$\n",
    "- $x_i^t$ : its word embedding\n",
    "\n",
    "Encoder:\n",
    "- $w_i^1,\\cdots,w_i^N$ be the words in sentence $s_i$ where N is the number of words in the sentence\n",
    "- At each time step, the encoder produces a hidden state $h_t^i$ which can be interpreted as the representation of the sequence $w_i^1,\\cdots,w_i^t$. The hidden state $h_i^N$ thus represents the full sentence.\n",
    "\n",
    "$r^t=\\sigma(W_r x^t + U_r h^{t-1})$ (1)\n",
    "\n",
    "$z^t=\\sigma(W_z x^t + U_z h^{t-1})$ (2)\n",
    "\n",
    "$\\bar{h}^r = tanh(Wx^t + U(r^t \\odot h^{t-1}))$ (3)\n",
    "\n",
    "$h^t=(1-z^t) \\odot h^{t-1} + z^t \\odot \\bar{h}^t$ (4)\n",
    "\n",
    "- $\\bar{h}^t$ is the proposed state update at time t\n",
    "- $z^t$ is the update gate\n",
    "- $r_t$ is the reset gate\n",
    "- $\\odot$ denotes a component-wise product\n",
    "- Both update gates takes values between zero and one\n",
    "\n",
    "Decoder:\n",
    "- $C_z$, $C_r$ and C that are used to bias the update gate, reset gate and hidden state computation by the sentence vector\n",
    "- One decoder is used for the next sentence $s_{i+1}$ while a second decoder is used for the previous sentence $s_{i-1}$\n",
    "- Separate parameters are used for each decoder with the exception of the vocabulary matrix V, which is the weight matrix connecting the decoder’s hidden state for computing a distribution over words\n",
    "- $h_{i+1}^t$: the hidden state of the decoder at time t\n",
    "\n",
    "$r^t=\\sigma(W_r^d x^{t-1} + U_r^d h^{t-1} + C_rh_i)$ (5)\n",
    "\n",
    "$z^t=\\sigma(W_z^d x^{t-1} + U_z^d h^{t-1} + C_zh_i)$ (6)\n",
    "\n",
    "$\\bar{h}^t = tanh(W^dx^{t-1} + U^d(r^t \\odot h^{t-1}) + Ch_i)$ (7)\n",
    "\n",
    "$h_{i+1}^t=(1-z^t) \\odot h^{t-1} + z^t \\odot \\bar{h}^t$ (8)\n",
    "\n",
    "Given $h_{i+1}^t$, the probability of word $w_{i+1}^t$ given the previous t − 1 words and the encoder vector is\n",
    "\n",
    "$P(w_{i+1}^t | w_{i+1}^{<t},h_i) \\propto exp(v_{w_{i+1}^t}h_{i+1}^t)$ (9)\n",
    "- $v_{w_{i+1}^t}$:the row of V corresponding to the word of $w_{i+1}^t$\n",
    "\n",
    "An analogous computation is performed for the previous sentence $s_{i-1}$.\n",
    "\n",
    "Objective:\n",
    "\n",
    "Given a tuple $(s_{i-1}, s_i, s_{i+1})$, the objective optimized is the sum of the log-probabilities\n",
    "for the forward and backward sentences conditioned on the encoder representation:\n",
    "\n",
    "$\\sum_t log P(w_{i+1}^t | w_{i+1}^{<t},h_i) + \\sum_t log P(w_{i-1}^t | w_{i-1}^{<t},h_i)$ (10)\n",
    "\n",
    "The total objective is the above summed over all such training tuples.\n",
    "\n",
    "### Vocabulary expansion\n",
    "Our goal is to construct a mapping f : $V_{w2v} \\to V_{rnn}$\n",
    "- $V_{w2v}$:the word embedding space of these word representations\n",
    "- $ V_{rnn}$:the RNN word embedding space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}