natural-language-processing/A-Latent-Semantic-Model-with-Convolutional-Pooling-Structure-for-Information-Retrieval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval\n",
    "\n",
    "    Yelong Shen,Xiaodong He,Jianfeng Gao,Li Deng,Grégoire Mesnil\n",
    "    CIKM’14, November 03 2014\n",
    "\n",
    "https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/cikm2014_cdssm_final.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "在DSSM基础上增加卷积层和Max-Pooling层"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EXTRACTING CONTEXTUAL FEATURES FOR IR USING CLSM\n",
    "### The CLSM Architecture\n",
    "1. a word-n-gram layer obtained by running a contextual sliding window over the input word sequence (i.e., a query or a document),\n",
    "1. a letter-trigram layer that transforms each word-trigram into a letter-trigram representation vector,\n",
    "1. a convolutional layer that extracts contextual features for each word with its neighboring words defined by a window, e.g., a word-n-gram,\n",
    "1. a max-pooling layer that discovers and combines salient word-n-gram features to form a fixed-length sentence-level feature vector,\n",
    "1. a semantic layer that extracts a high-level semantic feature vector for the input word sequence.\n",
    "\n",
    "![1](http://ou8qjsj0m.bkt.clouddn.com//17-8-6/1347804.jpg)\n",
    "\n",
    "### Letter-trigram based Word-n-gram Representation\n",
    "![2](http://ou8qjsj0m.bkt.clouddn.com//17-8-6/39329482.jpg)\n",
    "\n",
    "In Figure 1, the letter-trigram matrix $W_f$ denotes the transformation from a word to its letter-trigram count vector, which requires no learning.\n",
    "\n",
    "for the t-th word-n-gram at the word-n-gram layer, we have:\n",
    "\n",
    "$l_t=[f_{t-d}^T,...,f_t^T,...,f_{t+d}^T]^T, t=1,...,T\\ (1)$\n",
    "\n",
    "- $f_t$ is the letter-trigram representation of the t-th word\n",
    "- n=2d+1 is the letter-trigram representation of the t-th word\n",
    "\n",
    "### Modeling Word-n-gram-Level Contextual Features at the Convolutional Layer\n",
    "The convolution operation can be viewed as sliding window based feature extraction.\n",
    "\n",
    "$h_t=tanh(W_c \\cdot l_t), t=1,...,T$\n",
    "\n",
    "- $W_c$ is the feature transformation matrix, as known as the convolution matrix, that are shared among all word n-grams.\n",
    "\n",
    "![3](http://ou8qjsj0m.bkt.clouddn.com//17-8-6/874497.jpg)\n",
    "\n",
    "### Modeling Sentence-Level Semantic Features Using Max Pooling\n",
    "`Max pooling`:These local features need to be aggregated to obtain a sentence-level feature vector with a fixed size independent of the length of the input word sequence. Since many words do not have significant influence on the semantics of the sentence, we want to suppress the non-significant local features and retain in the global feature vector only the salient features that are useful for IR.\n",
    "\n",
    "$v(i)=max_{t=1,...,T}{ h_t(i)},i=1,...,K$\n",
    "\n",
    "- v(i) is the i-th element of the max pooling layer v, \n",
    "- $h_t(i)$ is the i-th element of the t-th local feature vector. \n",
    "- K is the dimensionality of the max pooling layer, which is the same as the dimensionality of the local contextual feature vectors ${ h_t }$.\n",
    "\n",
    "![4](http://ou8qjsj0m.bkt.clouddn.com//17-8-6/7996358.jpg)\n",
    "\n",
    "For each sentence, we examine the five most active neurons at the max-pooling layer, measured by v(i), and highlight the words in bold who win at these five neurons in the max operation (e.g., whose local features give these five highest neuron activation values). These examples show that the important concepts, as represented by these key words, make the most significant contribution to the overall semantic meaning of the sentence.\n",
    "\n",
    "### Latent Semantic Vector Representations\n",
    "After the sentence-level feature vector is produced by the max-pooling operation, one more non-linear transformation layer is applied to extract the high-level semantic representation, denoted by . \n",
    "\n",
    "$y=tanh(W_s \\cdot v)$\n",
    "\n",
    "- v is the global feature vector after max pooling,\n",
    "- $W_s$ is the semantic projection matrix, \n",
    "- y is the vector representation of the input query (or document) in the latent semantic space, with a dimensionality of L.\n",
    "\n",
    "### Using the CLSM for IR\n",
    "Given a query and a set of documents to be ranked, we first compute the semantic vector representations for the query and all the documents using the CLSM as described above.\n",
    "\n",
    "We compute the relevance score between the query and each document by measuring the cosine similarity between their semantic vectors.\n",
    "\n",
    "In Web search, given the query, the documents are ranked by their semantic relevance scores.\n",
    "\n",
    "## Learning the CLSM for IR\n",
    "We first convert the semantic relevance score between a query and a positive document to the posterior probability of that document given the query through softmax:\n",
    "\n",
    "$P(D^+|Q)=\\frac{exp(\\gamma R(Q,D^+))}{\\sum_{D' \\in D} exp(\\gamma R(Q,D'))}$\n",
    "\n",
    "In training, the model parameters are learned to maximize the likelihood of the clicked documents given the queries across the training set. That is, we minimize the following loss function\n",
    "\n",
    "$L(\\Lambda)=-log \\prod_{(Q,D^+)} P(D^+|Q)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}