week12/word_embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fun with Word Embeddings\n",
    "\n",
    "Today we are going to play with word embeddings: train our own little embedding, load one from a `gensim` model zoo and use it to visualize text corpora.\n",
    "\n",
    "This whole thing is going to happen on top of an embedding dataset.\n",
    "\n",
    "__Requirements:__  `pip install --upgrade nltk gensim bokeh` , but only if you're running locally. All of these are preinstalled in Colab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import numpy as np\n",
    "\n",
    "data_path = Path('./quora.txt')\n",
    "\n",
    "if not data_path.exists():\n",
    "    !wget https://www.dropbox.com/s/obaitrix9jyu84r/quora.txt?dl=1 -O $data_path\n",
    "    # alternative download link: https://yadi.sk/i/BPQrUu1NaTduEw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!head -n 20 $data_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with data_path.open(encoding='utf-8') as fp:\n",
    "    data = fp.readlines()\n",
    "\n",
    "data[50]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tokenization:__ a typical first step for an NLP task is to split raw data into words.\n",
    "The text we're working with is in raw format: with all the punctuation and smiles attached to some words, so a simple `str.split` won't do.\n",
    "\n",
    "Let's use __`nltk`__ - a library that handles many NLP tasks like tokenization, stemming or part-of-speech tagging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import WordPunctTokenizer\n",
    "tokenizer = WordPunctTokenizer()\n",
    "\n",
    "print(tokenizer.tokenize(data[50]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TASK: lowercase everything and extract tokens with tokenizer. \n",
    "# data_tok should be a list of lists of tokens for each line in data.\n",
    "\n",
    "data_tok = <YOUR CODE>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert all(isinstance(row, (list, tuple)) for row in data_tok), 'please convert each line into a list of tokens (strings)'\n",
    "assert all(all(isinstance(tok, str) for tok in row) for row in data_tok), 'please convert each line into a list of tokens (strings)'\n",
    "is_latin = lambda tok: all('a' <= x.lower() <= 'z' for x in tok)\n",
    "assert all(map(lambda l: not is_latin(l) or l.islower(), map(' '.join, data_tok))), 'please make sure to lowercase the data'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print([' '.join(row) for row in data_tok[:2]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Word vectors:__ as the saying goes, there's more than one way to train word embeddings. There's Word2Vec and GloVe with different objective functions. Then there's fasttext that uses character-level models to train word embeddings. \n",
    "\n",
    "The choice is huge, so let's start someplace small: __`gensim`__ is another NLP library that features many vector-based models incuding word2vec."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "from gensim.models import Word2Vec\n",
    "model = Word2Vec(\n",
    "    data_tok, \n",
    "    size=32,      # embedding vector size\n",
    "    min_count=5,  # consider words that occured at least 5 times\n",
    "    window=5).wv  # define context as a 5-word window around the target word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now you can get word vectors!\n",
    "model.get_vector('anything')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# or query similar words directly. Go play with it!\n",
    "model.most_similar('bread')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using pre-trained model\n",
    "\n",
    "Took it a while, huh? Now imagine training life-sized (100~300D) word embeddings on gigabytes of text: Wikipedia articles or Twitter posts. \n",
    "\n",
    "Thankfully, nowadays you can get a pre-trained word embedding model in 2 lines of code (no SMS required, promise)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim.downloader\n",
    "\n",
    "# Embeddings take a while to download, so we fetch a small (25D) model\n",
    "# List of available models: https://github.com/RaRe-Technologies/gensim-data\n",
    "# or simply try gensim.downloader.info()\n",
    "model = gensim.downloader.load('glove-twitter-25')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.most_similar(positive=['coder', 'money'], negative=['brain'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing word vectors\n",
    "\n",
    "One way to see if our vectors are any good is to plot them. Thing is, those vectors are in 20D+ space and we humans are more used to 2-3D.\n",
    "\n",
    "Luckily, we machine learners know about __dimensionality reduction__ methods.\n",
    "\n",
    "Let's use that to plot 1000 most frequent words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = sorted(\n",
    "    model.vocab.keys(), \n",
    "    key=lambda word: model.vocab[word].count,\n",
    "    reverse=True)[:1000]\n",
    "\n",
    "print(words[::100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For each word, compute its vector with the model\n",
    "word_vectors = <YOUR CODE>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert isinstance(word_vectors, np.ndarray)\n",
    "assert word_vectors.shape == (len(words), model.vector_size)\n",
    "assert np.isfinite(word_vectors).all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Linear projection: PCA\n",
    "\n",
    "The simplest linear dimensionality reduction method is __P__rincipial __C__omponent __A__nalysis.\n",
    "\n",
    "In geometric terms, PCA tries to find axes along which most of the variance occurs. The \"natural\" axes, if you wish.\n",
    "\n",
    "<img src='https://github.com/yandexdataschool/Practical_RL/raw/master/yet_another_week/_resource/pca_fish.png' style='width:30%'>\n",
    "\n",
    "\n",
    "Under the hood, it attempts to decompose object-feature matrix $X$ into two smaller matrices: $W$ and $\\hat W$ minimizing _mean squared error_:\n",
    "\n",
    "$$\\|(X W) \\hat{W} - X\\|^2_2 \\to \\min_{W, \\hat{W}}$$\n",
    "- $X \\in \\mathbb{R}^{n \\times m}$ - object matrix (**centered**);\n",
    "- $W \\in \\mathbb{R}^{m \\times d}$ - matrix of direct transformation;\n",
    "- $\\hat{W} \\in \\mathbb{R}^{d \\times m}$ - matrix of reverse transformation;\n",
    "- $n$ samples, $m$ original dimensions and $d$ target dimensions;\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# map word vectors onto 2d plane with PCA. Use good old sklearn api (fit, transform)\n",
    "# after that, normalize vectors to make sure they have zero mean and unit variance\n",
    "word_vectors_pca = <YOUR CODE>\n",
    "\n",
    "# and maybe MORE OF YOUR CODE here :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert word_vectors_pca.shape == (len(word_vectors), 2), 'there must be a 2d vector for each word'\n",
    "assert max(abs(word_vectors_pca.mean(0))) < 1e-5, 'points must be zero-centered'\n",
    "assert max(abs(1.0 - word_vectors_pca.std(0))) < 1e-2, 'points must have unit variance'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's draw it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bokeh.models as bm, bokeh.plotting as pl\n",
    "from bokeh.io import output_notebook\n",
    "\n",
    "output_notebook()\n",
    "\n",
    "def draw_vectors(x, y, radius=10, alpha=0.25, color='blue',\n",
    "                 width=600, height=400, show=True, **kwargs):\n",
    "    ''' draws an interactive plot for data points with auxilirary info on hover '''\n",
    "    if isinstance(color, str): color = [color] * len(x)\n",
    "    data_source = bm.ColumnDataSource({ 'x' : x, 'y' : y, 'color': color, **kwargs })\n",
    "\n",
    "    fig = pl.figure(active_scroll='wheel_zoom', width=width, height=height)\n",
    "    fig.scatter('x', 'y', size=radius, color='color', alpha=alpha, source=data_source)\n",
    "\n",
    "    fig.add_tools(bm.HoverTool(tooltips=[(key, '@' + key) for key in kwargs.keys()]))\n",
    "    if show: pl.show(fig)\n",
    "    return fig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "draw_vectors(word_vectors_pca[:, 0], word_vectors_pca[:, 1], token=words)\n",
    "\n",
    "# hover a mouse over there and see if you can identify the clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing neighbors with t-SNE\n",
    "PCA is nice but it's strictly linear and thus only able to capture coarse high-level structure of the data.\n",
    "\n",
    "If we instead want to focus on keeping neighboring points near, we could use TSNE, which is itself an embedding method. Here you can read __[more on TSNE](https://distill.pub/2016/misread-tsne/)__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.manifold import TSNE\n",
    "\n",
    "# map word vectors onto 2d plane with TSNE.\n",
    "# hint: don't panic it may take a minute or two to fit.\n",
    "# normalize them just like as with PCA\n",
    "word_tsne = <YOUR CODE>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "draw_vectors(word_tsne[:, 0], word_tsne[:, 1], color='green', token=words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing phrases\n",
    "\n",
    "Word embeddings can also be used to represent short phrases. The simplest way is to take __an average__ of vectors for all tokens in the phrase with some weights.\n",
    "\n",
    "This trick is useful to identify what data are you working with: find if there are any outliers, clusters or other artefacts.\n",
    "\n",
    "Let's try this new hammer on our data!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_phrase_embedding(phrase):\n",
    "    '''\n",
    "    Convert phrase to a vector by aggregating it's word embeddings. See description above.\n",
    "    '''\n",
    "    # 1. lowercase phrase\n",
    "    # 2. tokenize phrase\n",
    "    # 3. average word vectors for all words in tokenized phrase\n",
    "    # skip words that are not in model's vocabulary\n",
    "    # if all words are missing from vocabulary, return zeros\n",
    "    \n",
    "    vector = np.zeros([model.vector_size], dtype=np.float32)\n",
    "\n",
    "    <YOUR CODE>\n",
    "    \n",
    "    return vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vector = get_phrase_embedding(\"I'm very sure. This never happened to me before...\")\n",
    "\n",
    "assert np.allclose(\n",
    "    vector[::5],\n",
    "    np.array([ 0.1663555 ,  0.0201319 , -0.34534982,  0.1303215 ,  0.12880965],\n",
    "              dtype=np.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's only consider ~5k phrases for a first run.\n",
    "chosen_phrases = data[::len(data) // 1000]\n",
    "\n",
    "# compute vectors for chosen phrases\n",
    "phrase_vectors = <YOUR CODE>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert isinstance(phrase_vectors, np.ndarray) and np.isfinite(phrase_vectors).all()\n",
    "assert phrase_vectors.shape == (len(chosen_phrases), model.vector_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Map vectors into 2D space with PCA, T-SNE, or your other method of choice.\n",
    "# Don't forget to normalize.\n",
    "\n",
    "phrase_vectors_2d = <YOUR CODE>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "draw_vectors(phrase_vectors_2d[:, 0], phrase_vectors_2d[:, 1],\n",
    "             phrase=[phrase[:50] for phrase in chosen_phrases],\n",
    "             radius=20,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's build a simple 'similar question' engine with phrase embeddings we've built."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute vector embedding for all lines in data\n",
    "data_vectors = <YOUR CODE>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert isinstance(data_vectors, np.ndarray)\n",
    "assert np.isfinite(data_vectors).all()\n",
    "assert data_vectors.shape == (len(data), model.vector_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_nearest(query, k=10):\n",
    "    '''\n",
    "    given text line (query), return k most similar lines from data, sorted from most to least similar\n",
    "    similarity should be measured as cosine between query and line embedding vectors\n",
    "    Hint 1: it's okay to use global variables: data and data_vectors. see also: np.argpartition, np.argsort\n",
    "    Hint 2: it might be a good idea to normalize data_vectors only once rather than for each query\n",
    "    Hint 3: you will need to handle zero embedding vectors somehow\n",
    "    '''\n",
    "    \n",
    "    <YOUR CODE>\n",
    "    \n",
    "    return <YOUR CODE>  # top-k lines starting from most similar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = find_nearest(query='How do I enter the Matrix?', k=10)\n",
    "\n",
    "print(''.join(results))\n",
    "\n",
    "assert len(results) == 10 and isinstance(results[0], str)\n",
    "assert results[0] == 'How do I get to the dark web?\\n'\n",
    "assert results[3] == 'How do you print the gridlines in Excel 2007?\\n'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "find_nearest(query='How does Trump?', k=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "find_nearest(query=\"Why don't I ask a question myself?\", k=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Now what?__\n",
    "* Try running TSNE on all data, not just 1000 phrases\n",
    "* See what other embeddings are there in the model zoo: `gensim.downloader.info()`\n",
    "* Take a look at [FastText](https://github.com/facebookresearch/fastText) embeddings\n",
    "* Optimize find_nearest with locality-sensitive hashing: use [nearpy](https://github.com/pixelogik/NearPy) or `sklearn.neighbors`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acknowledgements\n",
    "\n",
    "This notebook is based on the [notebook for Seminar 1](https://github.com/yandexdataschool/nlp_course/blob/28a92e376f5229fe57f6e704c9f927909265b1e2/week01_embeddings/seminar.ipynb) in the YSDA NLP course."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}