natural-language-processing/Distributed-Representations-of-Sentences-and-Documents.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distributed Representations of Sentences and Documents\n",
    "\n",
    "    Quoc Le\n",
    "    Tomas Mikolov\n",
    "\n",
    "    2014 May 22\n",
    "\n",
    "https://arxiv.org/pdf/1405.4053.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "- Word Vector\n",
    "    - 目标\n",
    "        - 最大化average log probability：$\\frac{1}{T}\\sum_{t=k}^{T-k}logp(w_t|w_{t-k},\\cdots,w_{t+k})$\n",
    "    - 预测\n",
    "        - 多分类softmax：$p(w_t|w_{t-k},\\cdots,w_{w+k})=\\frac{e^{y_{w_t}}}{\\sum_i e^{y_i}}$\n",
    "        - 每个$y_i$是word的un-normalized log-probability：$y=b+Uh(w_{t-k},\\cdots,w_{t+k};W)\\ (1)$\n",
    "            - U，b是softmax参数\n",
    "            - h是concatenation\n",
    "- Paragraph Vector\n",
    "    - 每个paragraph被映射到矩阵D的一列\n",
    "    - 每个word被映射到矩阵W的一列\n",
    "    - 再做averaged或concatenated预测word\n",
    "- Paragraph Vector without word ordering\n",
    "    - 忽略上下文word，且预测的word必须从这段paragraph中选出"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Algorithms\n",
    "### Learning Vector Representation of Words\n",
    "Given a sequence of training words $w_1,w_2,w_3,\\cdots,w_T$, the objective of the word vector model is to maximize the average log probability\n",
    "\n",
    "$\\frac{1}{T}\\sum_{t=k}^{T-k}logp(w_t|w_{t-k},\\cdots,w_{t+k})$\n",
    "\n",
    "The prediction task is typically done via a multiclass classifier, such as softmax. There, we have\n",
    "\n",
    "$p(w_t|w_{t-k},\\cdots,w_{w+k})=\\frac{e^{y_{w_t}}}{\\sum_i e^{y_i}}$\n",
    "\n",
    "Each of $y_i$ is un-normalized log-probability for each output\n",
    "word i, computed as\n",
    "\n",
    "$y=b+Uh(w_{t-k},\\cdots,w_{t+k};W)\\ (1)$\n",
    "\n",
    "- U, b are the softmax parameters\n",
    "- h is constructed by a concatenation or average of word vectors extracted from W\n",
    "\n",
    "![1](http://ou8qjsj0m.bkt.clouddn.com//17-8-6/80388170.jpg)\n",
    "\n",
    "### Paragraph Vector: A distributed memory model\n",
    "In our Paragraph Vector framework (see Figure 2), every paragraph is mapped to a unique vector, represented by a column in matrix D and every word is also mapped to a unique vector, represented by a column in matrix W . The paragraph vector and word vectors are averaged or concatenated to predict the next word in a context.\n",
    "\n",
    "The only change in this model compared to the word vector framework is in equation 1, where h is constructed from W and D.\n",
    "\n",
    "![2](http://ou8qjsj0m.bkt.clouddn.com//17-8-6/3001285.jpg)\n",
    "\n",
    "In summary, the algorithm itself has two key stages: \n",
    "\n",
    "1. training to get word vectors W , softmax weights U, b and paragraph vectors D on already seen paragraphs;\n",
    "1. “the inference stage” to get paragraph vectors D for new paragraphs (never seen before) by adding more columns in D and gradient descending on D while holding W, U, b fixed.\n",
    "\n",
    "### Paragraph Vector without word ordering: Distributed bag of words\n",
    "Another way is to ignore the context words in the input, but force the model to predict words randomly sampled from the paragraph in the output. In reality, what this means is that at each iteration of stochastic gradient descent, we sample a text window, then sample a random word from the text window and form a classification task given the Paragraph Vector.\n",
    "\n",
    "![3](http://ou8qjsj0m.bkt.clouddn.com//17-8-6/2194169.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}