doc2vec.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doc2vec from scratch in PyTorch\n",
    "===============================\n",
    "\n",
    "Here we are implementing this useful algorithm with a library we know and trust. With luck this will be more accessible than reading the papers but more in-depth than typical \"install gensim and just do what I say\" tutorials, and still easy to understand for anyone whose maths skills have atrophied to nothing (like me). This is all based on the great work by [Nejc Ilenic](https://github.com/inejc/paragraph-vectors) and reading the referenced papers and gensim's source.\n",
    "\n",
    "`doc2vec` descends from `word2vec`, the basic form of which is that it is a model trained to predict the missing word in a context. Given sentences like \"the cat ___ on the mat\" it should predict \"sat\", and in doing so learn a useful representation of words. We can then extract the internal weights and re-use them as \"word embeddings\", vectors giving each word a position in N-dimensional space that is hopefully close to similar words and an appropriate distance from related words. \n",
    "\n",
    "`doc2vec` or \"Paragraph vectors\" extends the `word2vec` idea by simply adding a document id to each context. This helps the network learn associations between contexts and produces vectors that position each paragraph (document) in space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we need to load the data. We'll begin by overfitting on a tiny dataset just to check all the parts fit together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>In the week before their departure to Arrakis, when all the final scurrying about had reached a ...</td>\n",
       "      <td>[in, the, week, before, their, departure, to, arrakis, when, all, the, final, scurrying, about, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>It was a warm night at Castle Caladan, and the ancient pile of stone that had served the Atreide...</td>\n",
       "      <td>[it, was, a, warm, night, at, castle, caladan, and, the, ancient, pile, of, stone, that, had, se...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The old woman was let in by the side door down the vaulted passage by Paul's room and she was al...</td>\n",
       "      <td>[the, old, woman, was, let, in, by, the, side, door, down, the, vaulted, passage, by, paul, room...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>By the half-light of a suspensor lamp, dimmed and hanging near the floor, the awakened boy could...</td>\n",
       "      <td>[by, the, half, light, of, a, suspensor, lamp, dimmed, and, hanging, near, the, floor, the, awak...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                  text  \\\n",
       "0  In the week before their departure to Arrakis, when all the final scurrying about had reached a ...   \n",
       "1  It was a warm night at Castle Caladan, and the ancient pile of stone that had served the Atreide...   \n",
       "2  The old woman was let in by the side door down the vaulted passage by Paul's room and she was al...   \n",
       "3  By the half-light of a suspensor lamp, dimmed and hanging near the floor, the awakened boy could...   \n",
       "\n",
       "                                                                                                tokens  \n",
       "0  [in, the, week, before, their, departure, to, arrakis, when, all, the, final, scurrying, about, ...  \n",
       "1  [it, was, a, warm, night, at, castle, caladan, and, the, ancient, pile, of, stone, that, had, se...  \n",
       "2  [the, old, woman, was, let, in, by, the, side, door, down, the, vaulted, passage, by, paul, room...  \n",
       "3  [by, the, half, light, of, a, suspensor, lamp, dimmed, and, hanging, near, the, floor, the, awak...  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "pd.set_option(\"display.max_colwidth\", 100)\n",
    "\n",
    "example_df = pd.read_csv(\"data/example.csv\")\n",
    "\n",
    "def tokenize_text(df):\n",
    "    df[\"tokens\"] = df.text.str.lower().str.strip().apply(lambda x: [token.text.strip() for token in nlp(x) if token.text.isalnum()])\n",
    "    return df\n",
    "\n",
    "example_df = tokenize_text(example_df)\n",
    "\n",
    "example_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will need to construct a vocabulary so we can reference every word by an ID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset comprises 4 documents and 106 unique words (over the limit of 1 occurrences)\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "class Vocab:\n",
    "    def __init__(self, all_tokens, min_count=2):\n",
    "        self.min_count = min_count\n",
    "        self.freqs = {t:n for t, n in Counter(all_tokens).items() if n >= min_count}\n",
    "        self.words = sorted(self.freqs.keys())\n",
    "        self.word2idx = {w: i for i, w in enumerate(self.words)}\n",
    "        \n",
    "vocab = Vocab([tok for tokens in example_df.tokens for tok in tokens], min_count=1)\n",
    "\n",
    "print(f\"Dataset comprises {len(example_df)} documents and {len(vocab.words)} unique words (over the limit of {vocab.min_count} occurrences)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Words that appear extremely rarely can harm performance, so we add a simple mechanism to strip those out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>tokens</th>\n",
       "      <th>length</th>\n",
       "      <th>clean_tokens</th>\n",
       "      <th>clean_length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>In the week before their departure to Arrakis, when all the final scurrying about had reached a ...</td>\n",
       "      <td>[in, the, week, before, their, departure, to, arrakis, when, all, the, final, scurrying, about, ...</td>\n",
       "      <td>32</td>\n",
       "      <td>[in, the, week, before, their, departure, to, arrakis, when, all, the, final, scurrying, about, ...</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>It was a warm night at Castle Caladan, and the ancient pile of stone that had served the Atreide...</td>\n",
       "      <td>[it, was, a, warm, night, at, castle, caladan, and, the, ancient, pile, of, stone, that, had, se...</td>\n",
       "      <td>39</td>\n",
       "      <td>[it, was, a, warm, night, at, castle, caladan, and, the, ancient, pile, of, stone, that, had, se...</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The old woman was let in by the side door down the vaulted passage by Paul's room and she was al...</td>\n",
       "      <td>[the, old, woman, was, let, in, by, the, side, door, down, the, vaulted, passage, by, paul, room...</td>\n",
       "      <td>34</td>\n",
       "      <td>[the, old, woman, was, let, in, by, the, side, door, down, the, vaulted, passage, by, paul, room...</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>By the half-light of a suspensor lamp, dimmed and hanging near the floor, the awakened boy could...</td>\n",
       "      <td>[by, the, half, light, of, a, suspensor, lamp, dimmed, and, hanging, near, the, floor, the, awak...</td>\n",
       "      <td>53</td>\n",
       "      <td>[by, the, half, light, of, a, suspensor, lamp, dimmed, and, hanging, near, the, floor, the, awak...</td>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                  text  \\\n",
       "0  In the week before their departure to Arrakis, when all the final scurrying about had reached a ...   \n",
       "1  It was a warm night at Castle Caladan, and the ancient pile of stone that had served the Atreide...   \n",
       "2  The old woman was let in by the side door down the vaulted passage by Paul's room and she was al...   \n",
       "3  By the half-light of a suspensor lamp, dimmed and hanging near the floor, the awakened boy could...   \n",
       "\n",
       "                                                                                                tokens  \\\n",
       "0  [in, the, week, before, their, departure, to, arrakis, when, all, the, final, scurrying, about, ...   \n",
       "1  [it, was, a, warm, night, at, castle, caladan, and, the, ancient, pile, of, stone, that, had, se...   \n",
       "2  [the, old, woman, was, let, in, by, the, side, door, down, the, vaulted, passage, by, paul, room...   \n",
       "3  [by, the, half, light, of, a, suspensor, lamp, dimmed, and, hanging, near, the, floor, the, awak...   \n",
       "\n",
       "   length  \\\n",
       "0      32   \n",
       "1      39   \n",
       "2      34   \n",
       "3      53   \n",
       "\n",
       "                                                                                          clean_tokens  \\\n",
       "0  [in, the, week, before, their, departure, to, arrakis, when, all, the, final, scurrying, about, ...   \n",
       "1  [it, was, a, warm, night, at, castle, caladan, and, the, ancient, pile, of, stone, that, had, se...   \n",
       "2  [the, old, woman, was, let, in, by, the, side, door, down, the, vaulted, passage, by, paul, room...   \n",
       "3  [by, the, half, light, of, a, suspensor, lamp, dimmed, and, hanging, near, the, floor, the, awak...   \n",
       "\n",
       "   clean_length  \n",
       "0            32  \n",
       "1            39  \n",
       "2            34  \n",
       "3            53  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def clean_tokens(df, vocab):\n",
    "    df[\"length\"] = df.tokens.apply(len)\n",
    "    df[\"clean_tokens\"] = df.tokens.apply(lambda x: [t for t in x if t in vocab.freqs.keys()])\n",
    "    df[\"clean_length\"] = df.clean_tokens.apply(len)\n",
    "    return df\n",
    "\n",
    "example_df = clean_tokens(example_df, vocab)\n",
    "example_df[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The difficulty with our \"the cat _ on the mat\" problem is that the missing word could be any one in the vocabulary V and so the network would have |V| outputs for each input e.g. a huge vector containing zero for every word in the vocabulary and some positive number for \"sat\" if the network was perfectly trained. For calculating loss we need to turn that into a probabilty distribution, i.e. _softmax_ it. Computing the softmax for such a large vector is expensive.\n",
    "\n",
    "So the trick (one of many possible) we will use is _Noise Contrastive Estimation (NCE)_. We change our \"the cat _ on the mat\" problem into a multiple choice problem, asking the network to choose between \"sat\" and some random wrong answers like \"hopscotch\" and \"luxuriated\". This is easier to compute the softmax for since it's now a binary classifier (right or wrong answer) and the output is simply of a vector of size 1 + k where k is the number of random incorrect options.\n",
    "\n",
    "Happily, this alternative problem still learns equally useful word representations. We just need to adjust the examples and the loss function. There is a simplified version of the NCE loss function called _Negative Sampling (NEG)_ that we can use here.\n",
    "\n",
    "[Notes on Noise Contrastive Estimation and Negative Sampling (C. Dyer)](https://arxiv.org/abs/1410.8251) explains the derivation of the NCE and NEG loss functions.\n",
    "\n",
    "When we implement the loss function, we assume that the first element in a samples/scores vector is the score for the positive sample and the rest are negative samples. This convention saves us from having to pass around an auxiliary vector indicating which sample was positive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class NegativeSampling(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(NegativeSampling, self).__init__()\n",
    "        self.log_sigmoid = nn.LogSigmoid()\n",
    "    def forward(self, scores):\n",
    "        batch_size = scores.shape[0]\n",
    "        n_negative_samples = scores.shape[1] - 1   # TODO average or sum the negative samples? Summing seems to be correct by the paper\n",
    "        positive = self.log_sigmoid(scores[:,0])\n",
    "        negatives = torch.sum(self.log_sigmoid(-scores[:,1:]), dim=1)\n",
    "        return -torch.sum(positive + negatives) / batch_size  # average for batch\n",
    "\n",
    "loss = NegativeSampling()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's helpful to play with some values to reassure ourselves that this function does the right thing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>scores</th>\n",
       "      <th>loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[1, -1, -1, -1]</td>\n",
       "      <td>tensor(1.2530)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[0.5, -1, -1, -1]</td>\n",
       "      <td>tensor(1.4139)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[0, -1, -1, -1]</td>\n",
       "      <td>tensor(1.6329)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[0, 0, 0, 0]</td>\n",
       "      <td>tensor(2.7726)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[0, 0, 0, 1]</td>\n",
       "      <td>tensor(3.3927)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[0, 1, 1, 1]</td>\n",
       "      <td>tensor(4.6329)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[0.5, 1, 1, 1]</td>\n",
       "      <td>tensor(4.4139)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[1, 1, 1, 1]</td>\n",
       "      <td>tensor(4.2530)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              scores            loss\n",
       "0    [1, -1, -1, -1]  tensor(1.2530)\n",
       "1  [0.5, -1, -1, -1]  tensor(1.4139)\n",
       "2    [0, -1, -1, -1]  tensor(1.6329)\n",
       "3       [0, 0, 0, 0]  tensor(2.7726)\n",
       "4       [0, 0, 0, 1]  tensor(3.3927)\n",
       "5       [0, 1, 1, 1]  tensor(4.6329)\n",
       "6     [0.5, 1, 1, 1]  tensor(4.4139)\n",
       "7       [1, 1, 1, 1]  tensor(4.2530)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch \n",
    "\n",
    "data = [[[1, -1, -1, -1]],  # this dummy data uses -1 to 1, but the real model is unconstrained\n",
    "        [[0.5, -1, -1, -1]],\n",
    "        [[0, -1, -1, -1]],\n",
    "        [[0, 0, 0, 0]],\n",
    "        [[0, 0, 0, 1]],\n",
    "        [[0, 1, 1, 1]],\n",
    "        [[0.5, 1, 1, 1]],\n",
    "        [[1, 1, 1, 1]]]\n",
    "\n",
    "loss_df = pd.DataFrame(data, columns=[\"scores\"])\n",
    "loss_df[\"loss\"] = loss_df.scores.apply(lambda x: loss(torch.FloatTensor([x])))\n",
    "\n",
    "loss_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Higher scores for the positive sample (always the first element) reduce the loss but higher scores for the negative samples increase the loss. This looks like the right behaviour."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With that in the bag, let's look at creating training data. The general idea is to create a set of examples where each example has:\n",
    "\n",
    "- doc id\n",
    "- sample ids - a collection of the target token and some noise tokens\n",
    "- context ids - tokens before and after the target token\n",
    "\n",
    "e.g. If our context size was 2, the first example from the above dataset would be:\n",
    "\n",
    "```\n",
    "{\"doc_id\": 0,\n",
    " \"sample_ids\": [word2idx[x] for x in [\"week\", \"random-word-from-vocab\", \"random-word-from-vocab\"...],\n",
    " \"context_ids\": [word2idx[x] for x in [\"in\", \"the\", \"before\", \"their\"]]}\n",
    " ```\n",
    " \n",
    " The random words are chosen according to a probability distribution:\n",
    " \n",
    " > a unigram distribution raised to the 3/4rd power, as proposed by T. Mikolov et al. in Distributed Representations of Words and Phrases and their Compositionality\n",
    "\n",
    "This has the effect of slightly increasing the relative probability of rare words (look at the graph of `y=x^0.75` below and see how the lower end is raised above `y=x`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.vegalite.v2+json": {
       "$schema": "https://vega.github.io/schema/vega-lite/v2.6.0.json",
       "config": {
        "view": {
         "height": 300,
         "width": 400
        }
       },
       "data": {
        "name": "data-afc3dc0951a9e12875119a5af86e52b5"
       },
       "datasets": {
        "data-afc3dc0951a9e12875119a5af86e52b5": [
         {
          "x": 0,
          "y": 0
         },
         {
          "x": 0.01,
          "y": 0.03162277660168379
         },
         {
          "x": 0.02,
          "y": 0.053182958969449884
         },
         {
          "x": 0.03,
          "y": 0.07208434242404263
         },
         {
          "x": 0.04,
          "y": 0.08944271909999159
         },
         {
          "x": 0.05,
          "y": 0.10573712634405642
         },
         {
          "x": 0.06,
          "y": 0.12123093028059741
         },
         {
          "x": 0.07,
          "y": 0.13608915892697748
         },
         {
          "x": 0.08,
          "y": 0.15042412372345573
         },
         {
          "x": 0.09,
          "y": 0.16431676725154984
         },
         {
          "x": 0.1,
          "y": 0.1778279410038923
         },
         {
          "x": 0.11,
          "y": 0.19100490227716513
         },
         {
          "x": 0.12,
          "y": 0.2038853093816547
         },
         {
          "x": 0.13,
          "y": 0.2164998073464082
         },
         {
          "x": 0.14,
          "y": 0.22887377179317683
         },
         {
          "x": 0.15,
          "y": 0.2410285256833955
         },
         {
          "x": 0.16,
          "y": 0.25298221281347033
         },
         {
          "x": 0.17,
          "y": 0.26475044029330763
         },
         {
          "x": 0.18,
          "y": 0.2763467610958144
         },
         {
          "x": 0.19,
          "y": 0.28778304315451386
         },
         {
          "x": 0.2,
          "y": 0.29906975624424414
         },
         {
          "x": 0.21,
          "y": 0.3102161981490854
         },
         {
          "x": 0.22,
          "y": 0.32123067524150845
         },
         {
          "x": 0.23,
          "y": 0.33212064831351956
         },
         {
          "x": 0.24,
          "y": 0.34289285156385596
         },
         {
          "x": 0.25,
          "y": 0.3535533905932738
         },
         {
          "x": 0.26,
          "y": 0.3641078238014289
         },
         {
          "x": 0.27,
          "y": 0.37456123052590357
         },
         {
          "x": 0.28,
          "y": 0.38491826849295824
         },
         {
          "x": 0.29,
          "y": 0.39518322257770583
         },
         {
          "x": 0.3,
          "y": 0.4053600464421103
         },
         {
          "x": 0.31,
          "y": 0.41545239829339137
         },
         {
          "x": 0.32,
          "y": 0.42546367175559907
         },
         {
          "x": 0.33,
          "y": 0.43539702265375557
         },
         {
          "x": 0.34,
          "y": 0.4452553923589699
         },
         {
          "x": 0.35000000000000003,
          "y": 0.45504152822405847
         },
         {
          "x": 0.36,
          "y": 0.46475800154489
         },
         {
          "x": 0.37,
          "y": 0.47440722340731084
         },
         {
          "x": 0.38,
          "y": 0.4839914587188715
         },
         {
          "x": 0.39,
          "y": 0.4935128386754873
         },
         {
          "x": 0.4,
          "y": 0.5029733718731741
         },
         {
          "x": 0.41000000000000003,
          "y": 0.5123749542422491
         },
         {
          "x": 0.42,
          "y": 0.5217193779544038
         },
         {
          "x": 0.43,
          "y": 0.5310083394307343
         },
         {
          "x": 0.44,
          "y": 0.5402434465602292
         },
         {
          "x": 0.45,
          "y": 0.549426225222706
         },
         {
          "x": 0.46,
          "y": 0.5585581251971565
         },
         {
          "x": 0.47000000000000003,
          "y": 0.5676405255254853
         },
         {
          "x": 0.48,
          "y": 0.576674739392341
         },
         {
          "x": 0.49,
          "y": 0.5856620185738529
         },
         {
          "x": 0.5,
          "y": 0.5946035575013605
         },
         {
          "x": 0.51,
          "y": 0.6035004969804791
         },
         {
          "x": 0.52,
          "y": 0.6123539276009055
         },
         {
          "x": 0.53,
          "y": 0.6211648928681236
         },
         {
          "x": 0.54,
          "y": 0.629934392084505
         },
         {
          "x": 0.55,
          "y": 0.6386633830041155
         },
         {
          "x": 0.56,
          "y": 0.6473527842827909
         },
         {
          "x": 0.5700000000000001,
          "y": 0.6560034777426358
         },
         {
          "x": 0.58,
          "y": 0.6646163104680073
         },
         {
          "x": 0.59,
          "y": 0.6731920967482075
         },
         {
          "x": 0.6,
          "y": 0.6817316198804996
         },
         {
          "x": 0.61,
          "y": 0.6902356338456498
         },
         {
          "x": 0.62,
          "y": 0.6987048648669424
         },
         {
          "x": 0.63,
          "y": 0.7071400128625219
         },
         {
          "x": 0.64,
          "y": 0.7155417527999327
         },
         {
          "x": 0.65,
          "y": 0.7239107359608682
         },
         {
          "x": 0.66,
          "y": 0.7322475911233668
         },
         {
          "x": 0.67,
          "y": 0.7405529256680135
         },
         {
          "x": 0.68,
          "y": 0.7488273266140879
         },
         {
          "x": 0.6900000000000001,
          "y": 0.7570713615910638
         },
         {
          "x": 0.7000000000000001,
          "y": 0.7652855797503655
         },
         {
          "x": 0.71,
          "y": 0.7734705126218591
         },
         {
          "x": 0.72,
          "y": 0.7816266749191567
         },
         {
          "x": 0.73,
          "y": 0.7897545652974598
         },
         {
          "x": 0.74,
          "y": 0.7978546670673515
         },
         {
          "x": 0.75,
          "y": 0.8059274488676564
         },
         {
          "x": 0.76,
          "y": 0.8139733653002305
         },
         {
          "x": 0.77,
          "y": 0.8219928575293057
         },
         {
          "x": 0.78,
          "y": 0.829986353847804
         },
         {
          "x": 0.79,
          "y": 0.837954270212839
         },
         {
          "x": 0.8,
          "y": 0.8458970107524514
         },
         {
          "x": 0.81,
          "y": 0.8538149682454624
         },
         {
          "x": 0.8200000000000001,
          "y": 0.8617085245761865
         },
         {
          "x": 0.8300000000000001,
          "y": 0.869578051165608
         },
         {
          "x": 0.84,
          "y": 0.8774239093805121
         },
         {
          "x": 0.85,
          "y": 0.8852464509219427
         },
         {
          "x": 0.86,
          "y": 0.8930460181942644
         },
         {
          "x": 0.87,
          "y": 0.9008229446560111
         },
         {
          "x": 0.88,
          "y": 0.9085775551536168
         },
         {
          "x": 0.89,
          "y": 0.9163101662390513
         },
         {
          "x": 0.9,
          "y": 0.9240210864723069
         },
         {
          "x": 0.91,
          "y": 0.9317106167096201
         },
         {
          "x": 0.92,
          "y": 0.9393790503782488
         },
         {
          "x": 0.93,
          "y": 0.9470266737385726
         },
         {
          "x": 0.9400000000000001,
          "y": 0.9546537661342305
         },
         {
          "x": 0.9500000000000001,
          "y": 0.9622606002309622
         },
         {
          "x": 0.96,
          "y": 0.9698474422447793
         },
         {
          "x": 0.97,
          "y": 0.9774145521600454
         },
         {
          "x": 0.98,
          "y": 0.9849621839380145
         },
         {
          "x": 0.99,
          "y": 0.992490585716335
         }
        ]
       },
       "encoding": {
        "x": {
         "field": "x",
         "type": "quantitative"
        },
        "y": {
         "field": "y",
         "type": "quantitative"
        }
       },
       "mark": "line",
       "title": "x^0.75"
      },
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcoAAAFxCAYAAADzkehqAAAgAElEQVR4Xu2dC5RlVXnnf6e6oRpXjGSUiMYHUMD4ttGAkvHRLiX46JZqtYFSg9UYzSwSQmkMFI5OV+tI3cAk3QxJBkYTaBND4wMaq2Am0SAsW4zigwZCOkYHmlHA+IBePqCBrjNrV52ib926j7PPt8+9d9/zv2uxatF3//fZ+/d95/zvPmefvRP0EQEREAEREAERaEkgERsREAEREAEREIHWBGSUyg4REAEREAERaENARqn0EAEREAEREAEZpXJABERABERABIoR0IiyGDepREAEREAEKkJARlmRQKubIiACIiACxQjIKItxk0oEfAm8H/hT4BfAbwB76yp4ObAFeAHwfeC/AZ9qcoCJrFzjV9uBs4CfNtGMAe57fURABAoSkFEWBCeZCHgQcOfZvwL/ALwDmAIuzvSrgLuA7wDrgY8A/zkzzd0djjEKXAOcDnwd+L/AVuB9Hm1TUREQgQ4EZJRKERGwE/gj4L8DbtToRoavBm4ErgTeDvw28H+A/wicDZwMPAdIgTcA12cjwv8JvAz4J2BTZpqtWvdE4F8yk30lcBzwLWBzZsT2XqkGERCBeQIySiWCCNgJuPPoi8DxwIsy4/tV4IXAA8AO4FeA1wHPA/4ZOCnTnJONAt8KXA08G7gb+Lts9NmqdR8FPgS8Cvgy8BrgBuBrwDOBg4DPAc7Ef2nvomoQgeoSkFFWN/bqeVgCzpxuA/YBv56NGr+QmZa7teqeFX4mO+RN2fNEd6v1fOAC4M3ADPA04F7gWsDdWm32eUo2knQjSDd6dZ/F27BuJOvqdc8sPwbUsmOE7a1qE4EKEZBRVijY6mrpBNxzxz/Mnhe6W6ju40Z+7wWeATya/Zt7pvi3wJGZqTld44jSTeZ5Z4sWL07q2Qhc0aLMIdnEoVuBl5Tecx1ABAaYgIxygIOrrnWVgJuxeks20ntudtvU3T7t9HHPL/++7hnlicDNwAeB6Rbir2TPMp8K/KRFmSdkRunq+k+dGqHvRUAEWhOQUSo7RMBOYDgbRT45m63qbpuuBl6cPW9sd4SDge9l/7nbp38CvCub7ON07ratM9LX11Xinjn+KHueufjP48BfABcCFwEfyCb2uGeg/8PeRdUgAtUlIKOsbuzV83AE/ix7JeMU4PPAscAu4JvZM8T9HQ71m5nJPT97xcNN0nH1HNHEKA8D/j0bddaPFFdkt3ndLNvDgf8HfCIzzblwXVVNIlA9AjLK6sVcPRYBERABEfAgIKP0gKWiIiACIiAC1SMgo6xezNVjERABERABDwIySg9YKioCIiACIlA9AjLK6sVcPRYBERABEfAgIKP0gKWiIiACIiAC1SMgo6xezNVjERABERABDwIySg9YKioCIiACIlA9AjLK6sVcPRYBERABEfAgIKP0gKWiIiACIiAC1SPQD0bpFoHeBrhlwC5tEgLXRrcZrlvL8n7gzGz5rupFSz0WAREQARHoOoFeG6VbENqZoNvc9vstjNLtree2FXJ/3S7uzkyP6TopHVAEREAERKCSBHptlIvQ3SLQP25hlG4nhDuyUacr7zbHPRm4r5IRU6dFQAREQAS6SiAGo7ws2/l9NiNzQ7Z33+7p6empJEk21RM77LDDWLNmTVch6mAiIAIiIAKDQWBkZGSZL8ZglG5/vTvrdnK/HTgpe165LDK1Wi2dnJzsl34ta9/3vve9tFkg+iXF1D5bJMRP/GwEbGrlXz5+o5uuOXT/vodHZ2pjV9QrWvHrF0Npd+t1HXAu4Pb6Ox6oZc8qmxKRUeZLlFaldKKJn42ATa38Ez8bgfZqZ5CP7XvkHNJ0goRDV5IeuaM2dveiql+N8hXAlxu65jamXQnsBI4GFme9bgTuBc4AbmmFQ0ZpSzNdqMTPRsCmVv6Jn41Aa/Wbzr/qXaTpVLKwITopXHsQ6UQMRhmciYzShlQXKvGzEbCplX/iZyOwXL128so1MD+XZXHyyi5IJ2ZrYzc2lu7XEWVoJsgobUh1oRI/GwGbWvknfjYCB9Sjk1ce8SjJpmThHXz32ZOQTjU+l6w/nowyFH1jPboQ2ACKn/jZCNjUyr/+5/f4c0jSqay1eyHdunJ41dYdm9c/2K4HMkpbfIOpdaLZUIqf+NkI2NTKv/7m555DJnPpVjdRx7U0Id22Aqbqn0PKKN2UWL0eYspkXQhM+BA/8bMRsKmrmn/zzyHTZAsJqxcIpjcBU82eQ8ooZZS2swx0oTcSrOqFyojtcbn42UhWjZ97DvnYgkGOZuQ6PoeUUcoobWeZjFL8zARsFVTtQm+jtVxdFX6tnkPO1sYWn0sWQqtnlIWwhRdVJZHDk1uoUfxsZMVP/GwEbOoQ+df0OeTwqolOE3XytFxGmYdSF8qESJQym6n22eiKn/jZCNjUg5x/zZ5Dzg2tmLj+glNvtVE7oJZRhiJprGeQE9mIJpdc/HJhallI/MTPRsCmLpJ/TZ9DJsnEzPRpO2ytyX/rul/Weg3WX816taEsksi2I/qp1T4/Xo2lxU/8bARsap/8K+s5ZLseaERpi28wtU+iBDuoR0VqnwesJkXFT/xsBGzqQcm/Mp9Dyig169V2lmmyjPiZCdgqGJQLvY1CcXXs/LrxHFJGKaMsfoZlythPNDMAYwXiZwMoftXk183nkDJKGaXtLNOIUvzMBGwVyCirxa8XzyFllDJK21kmoxQ/MwFbBTLK6vDr1XNIGaWM0naWySjFz0zAVoGMcvD5nfPxr7+mcV3W0O9DFqWoWa9FyQXW6UJgAyp+4mcjYFMr/4rzc88hX3rsYXd97Ts/XqzEtC5r8Za0Vsooy6BaoE6daAWg1UnET/xsBGxq5Z8/v8efQ6bpRLb9Ve79If2PZlPIKG38gql1otlQip/42QjY1Mo/P37uOSRpOpXAEU55wrFP4Vvf+dGRefeH9DuavbSM0s4wSA060WwYxU/8bARsauVfPn5v/OCnVw/NzW0B1mSKXZBOXPyeE740MjLStyvCySjzxbf0UjrRbIjFT/xsBGxq5V97fu4266P79m1JYDwruTchnZipjV3h/j9Wfn3r7EXTWWu9FiW3oIs1kW29DqcWPxtL8YuX39rztp8DTGXPIYF088rhVVvrt7+KNb69Nkp3fDc8d78+7gfOBG5uSJUVwMeBtwL3AL8DtNxWRUYZ74mWp+Wxnmh5+taNMuJnoyx+y/m5ZedSkssXn0NCetNKGG/2HDJWfr02yvXABOD+HgdcChzTEIpTgXcB7wDeAJwN/FardJdR6kJgI2BTx3ohsPU6nFr8bCy7yW9+2TmSy+ueQ+6BdHy2NnZjq150s31FSPbrM8qLgDuAbVmnbgNOBu6r6+Qm4AfAJ7J/+yHwdGB/MxAyyiLpcUATayLbeh1OLX42luLX//wsy87FGt9ejygvA2aA2Sw9bgDOAnbXpcvbs9HkGHBiVv7XgQenp6enkiRxRrrks2HDBlu2SS0CIiACIrCMwNe/82Ou/uo9PPTIwjjFve7xlhOfxSEHuydkg/FpNiu310Z5IXAnMD8jCrgdOCl7XrlI/SDgU8AbgauBVwBHtQqJRpS2ZI31F5+t1+HU4mdjKX79yW9++yvmByXZ6x7pTW7iTrvbrM16Emt8e22U64BzgVOA491SrdmzynrGzwBeCnwRcKNL95zyLTJK2wnVSh1rIpdDw79W8fNnVq8Qv/7i1+l1D9/WxhrfXhvl4qzXjcC9wBnALcDhwE7gaOBXgc8DLwO+kd2GdbNfm340ovRN3aXlY01kW6/DqcXPxlL8+odfntc9fFsba3x7bZS+nDuWl1F2RNS2QKyJbOt1OLX42ViKX+/5+bzu4dvaWOMro/SNtLF8rIli7HYwufjZUIqf+LUiMP+6R5psIWE0K9PxdQ9fmrHmn4zSN9LG8rEmirHbweTiZ0MpfuLXSKDV7h6ztbEpG63l6ljzT0YZOhM61BdronQZU8vDiZ8tEuInfvUE1p1/1ehcmrq1Wed390hIt60YXjVRv+ycjdhSdaz5J6MMmQU56oo1UXJ0rStFxM+GWfzEzxFotbuH7+sevjRjzT8ZpW+kjeVjTRRjt4PJxc+GUvyqzc/dZt3/8L5NaTK/dKj7LNndw0anszrW/JNRdo5t0BKxJkpQCIbKxM8AT7vD2OBFzs9topzMpVsP7O7BxSuHh6fKus3aDHas56+M0nzq+FUQa6L49bK80uJnYyt+1ePXbFWduaEVE9dfcGrLXZhslFqrY80/GWVZGdGi3lgTpcuYWh5O/GyREL/q8Guyqs6ehHRqcRNlG4li6ljzT0ZZLN6FVbEmSuEOBxaKnw2o+FWDXxmr6tjILahjzT8ZZYjoe9QRa6J4dLHUouJnwyt+g83v73feml4ys/tWElYv9LT1Jso2EsXUseafjLJYvAurYk2Uwh0OLBQ/G1DxG0x+TW+zJsnEzPRpO2w9DquONf9klGHzoGNtsSZKx451qYD42UCL3+Dxq7/N6vaFfOiRxzavHF61tZuzWfNSjTX/ZJR5IxyoXKyJEqj75mrEz4ZQ/AaHX+Pi5SlcO3X6i045fvXz+va6Hmv+9S3Qoums3UOKklvQxZrItl6HU4ufjaX4debXbvFy8evMr12JVvxklDau3molsjeyJQLxEz8bAZu61/m3dvKqTaTpRLZowF5It9YvXt7r9nWiG2v7ZJSdIhv4+1gTJTCGwtWJX2F0umNgQ9dTfs1usx5EOrGjNnZ3fbd0ftiCrBGljV8wtRLZhlL8xM9GwKbudv757hHZ7fb50oy1fRpR+kbaWD7WRDF2O5hc/GwoxS8efp1uszbrieJbTnxllDau3molsjeyJQLxEz8bAZu6G/mX9zarjNIWSx9+MsrwrNvW2I0TzdIltc9CT7OGbfSqzc/3NqvPhd4al1D6WK8vMspQGZCznlgTJWf3Si8mfjbE4tef/IrcZpVR2mLpw09GGZ61RpQlMtWF3gZX/PqLn+U2q8+F3tbrcOpY86/XRumOvwUYB+4HzgRubgjLQcDlwJuBH8H8ztwzrUKnBQdsSR1rItt6HU4tfjaWVeHn1mZ97OF9l5MwmhHbA+n4bG3sRgvBqvCzMGqn7dfXQ9Znxuf+HgdcChzT0JG3Ab8DvAM4FvgscJSMspxU0Ylm4yp+4teJQMMWWMsWDeikL3Kht9QZUhvr+dHrEeVFwB3AtiwYtwEnA/fVBec1wPuBtwNHA5cBJ8goQ6bvgbpiTeRyaPjXKn7+zOoVg8zP3WYlTbYsboHl1mZttmiAheAg87Nwyavt1xGlMz13G3U268gNwFnA7rqOOTN3W8W4W6/7gLcA17vvp6enp5Ik2dQIYcOGDXm5qJwIiIAIlErgoUf2c81X7+Fr3/nx/HH+w68czDvWHMXRT3tiqcdV5cUIjIyMLBtA9npEeSFwJ3BF1qXbgZOy55WLvXxPdlv2A8DTMpN8MfBwMwx6RlksORZV+kUqfjYCNvWg5V/DbVa3kXKpW2ANGj9bNvmr+3VEuQ44FzgFOB6oZaZY38MPAU8HnFEenk32cc8xfyaj9E+ETgqdaJ0Itf9e/MTPEXjjBz+9emhuzk1UXLNAJL1pJYw3rs1qo7VcrfyzEe1Xo1yc9boRuBc4A7glM8Sd2TPJpwB/C7wS2AtcAPx5KxwaUZaTKLZaw6l1IbCxFL9y+bnZrPsf3rcpTeZn57vPniRJJmamT3OPj0r/KL42xP1qlLZeNVHLKG1IdaKJn42ATR1z/r3p/KvelcylW7MtsEq/zdqMdMz8bJkTRi2jDMPRXIsS2YZQ/MTPRsCmbpZ/80vPkbh3vR+/zTo3tGLi+gtOvdV2NH+1zg9/ZvUKGaWNXzC1EtmGUvzEz0bApq7Pv/lFA/Y9cg6kU1mtexPSiZna2OLkRNvBCqh1fhSAVieRUdr4BVMrkW0oxU/8bARs6sX8a1x6Drh45fDw1I7N6x+0HcGm1vlRDr9evx5i61UTtZ5R2pDqRBM/GwGbut/z75Zb70w3X3nbjrql53ZBOmFdes5G7YC63/nF2j4ZZagMzVlPrImSs3ulFxM/G2LxK87P7fBxyMFDU24BATcDP02Yum769K3FawyvVHxtTHXr1cYvmFqJbEMpfuJnI+Cv7sbSc/6taq7Q+WEjKaO08QumViLbUIqf+NkI5Fe7yTqP7tu3JVnY3ch99py99jnPPvkVq/v2TpzOj/zxbVZSRmnjF0ytRLahFD/xsxHIp272TuRsbWxK+ZePX6tSsfLr219GRcOhyTxFyS3oYk1kW6/DqcXPxrLX/Jq9E1m/9Fyv29eJrtrXiVD77zWitPELplYi21CKn/jZCDRX530nUvlnox8rP40obXH3VseaKN4dLUkgfjaw4recn887keJXzfyTUdri7q3WieaNbIlA/MTPRuCAen4U+fC+y33eiVT+2ejHyk9GaYu7tzrWRPHuaEkC8bOBFb8Ffg37RO6FdKubrNOJrvh1ItT++1j5yShtcfdWx5oo3h0tSSB+NrBV59e4T2QK1x5EOpF3n8iq87NlX7yTBWWU1sh76nWieQJrKC5+4leEQJPJOoX2iVT+FaF/QBMrPxmlLe7e6lgTxbujJQnEzwa2ivx8Jut0oltFfp2Y+HwfKz8ZpU+UA5SNNVECdD1IFeJnw1glfs0m68wNDY1b9omsEj9bpjVXx8pPRllGNrSpM9ZE6TKmlocTP1skqsKvcbJOqAXMq8LPlmWt1bHyk1GWlREt6o01UbqMSUZZEvBBzz/rZJ1O2AedX6f+W7+PlZ+M0hp5T32sieLZzdKKi58N7aDyCzVZpxPdQeXXqd+hvo+Vn4wyVAbkrCfWRMnZvdKLiZ8N8SDyCzlZpxPdQeTXqc8hv4+Vn4wyZBbkqCvWRMnRta4UET8b5kHiV8ZknU50B4lfp76W8X2s/HptlO74W1jY7+1+4Ezg5oYAfQj4aMO/Hep2GG8WSO0eYkvvWBPZ1utwavGzsczLr2EbrNwr69haF+8L89Z+h9LnjW+o4/nW06p9vTbK9cAE4P4eB1wKHNOmc88B/gQ4pVUZGaVvaiwtH2si23odTi1+Npad+HXaBst29M7qTu3rXEO5JdQ+G99+NcqLgDuAbVn3bgNOBu5r0d0vAO8F7pJR2hKilVonmo2r+JXHb+3kVZsgXVyPdW+SJOMz06ftsB3RT634+vFqLB0rv16PKC8DZoDZDOgNwFnA7ibhWAu8LhuBzn89PT09lSTJpsayGzZssEVTahEQgb4h8N37fsbVX72HH/zkl/NtOuHYp/CWE5/FIQev6Js2qiGDQ2BkZGSZL/baKC8E7gSuyDDfDpyUPa9sJH898P4WJvp4Wd16tSVsrL/4bL0OpxY/G8t6fm6yzv6H921Kk/nHM+6zB9Lx2drYjbajFFcrvsXZOWWs/HptlOuAc7NnjscDtexZZWM0ngDs6vD8cl4jo6xmItt6HU4d64UgHAFbTYv81p1/1ehcmm5J4IiFGtPNebbBsh29s1rx7cyoXYlY+fXaKBdnvW4E7gXOAG4BDgd2Akdn0J+bTfR5dacwySg7EWr/fayJbOt1OLX42Vje8S/fSSev+NaOA5sppzfNDa2YsKzPamvRUrXia6MZK79eG6WNehO1jNKGNNZEtvU6nFr8irN067MeMrxi60OP7HeV7A21PmvxFi1XKr42mrHyk1Ha4u6tjjVRvDtakkD8bGD7kV/jKx++mynbiPip+5FffQ/UPr94Npbu19dDbL3SiDI4P51oNqTi58dv/pWPNJ0gYX4Rkd896egnjb72N/v2B7zi6xffvEZkqzWcWkYZjqWpJp1oJnzRzpqz9Tqcul/yb36Xj/1zl5Ow2vUuId22YnjVxJ+e8aIHmk3PD0fAVlO/8GvVC7WvnPj27S+3ot3VM8qi5BZ0OtHEz0agvbrZLh/1r3wo/2z0xa8cfjJKG1dvtRLZG9kSgfjFyy/PLh+Kb7zxzdPyWOMro8wT3YBlYk2UgAhMVYmfCV9P7hg0WThg19zQ0HizVz4U3/ji69PiWOMro/SJcoCysSZKgK4HqUL8bBi7zc8tHJDOpe5ZpJus03HhgG63z5em2udLbGn5WPnJKG1x91bHmijeHS1JIH42sN3it3yvyPSmlTC+ozZ2d7sedKt9RSmqfUXJLehi5SejtMXdWx1ronh3tCSB+NnAdoNf416RPgsHdKN9FoJqn4WejNJGL6Bas15tMHUhEL+iBELsFan8K0o/7hGbrdfh1K3yTyPKcIxz1aQLQS5MLQuJX3/yc8vPAVOLCwcU3StS8e3P+NpadUAda3xllKEyIGc9sSZKzu6VXkz8bIhD82u6/Nzw8PiOzesfLNLS0O0r0oZ2GrXPRjRWfjJKW9y91bEmindHSxKInw1sSH7zy8+RTmUt2lt0FFnfo5Dts5Fqrlb7bFRj5SejtMXdWx1ronh3tCSB+NnAhuDXuPwccPHK4eGpoqNIGaUtpuJXPj8ZZTjGuWoKcaHKdaCChdS+guAy2aDzaxhF7qlffs5GbkE96PxCMNKt4fIoajJPeWy9ataFwAvXssLi1xt+ZY4iNSKyxVT8yuenEWU4xrlq0oU+F6aWhcSvu/w6LWJua81yteJrIyp+5fCTUdq4equVyN7IlgjEr3v8li9inm5eObxqa4hnka16ofh2L762IxVTxxpfGWWxeBdWxZoohTscWCh+NqB5+PksYm5rjUaU4heagK0+PaO08QumznOhCnawAhWpfQWg1Uli59dsFDlbG1t8BcQGJ4c6dn45ulhqEfGz4ZVR2vgFUyuRbSjFrxx+vRxF1vdI8S0nvrZaw6ljja9uvYbLgVw1xZoouTrXhULiZ4PcjF+vR5EySltMxa98fr02Snf8LcA4cD9wJnBzk26/C/gY8ARgE3BJKzRaFN2WNDKi6vDrl1GkLvS2nBO/8vn12ijXAxOA+3sccClwTEO3Dwe+AKwDfgEcBXxNRhkuOXSihWMZyw+NfhpFKv+ql3/hehy2pn59RnkRcAewLevubcDJwH113X8n8Cbg5cAq4GzgszLKsAmyWFssF/pyem+vtd/53fEv30nPv+JbW9Nk/geq++yaGxoav/6CU2+1995eQ7/zU/tsMY6VX69HlJcBM8Bshv8G4Cxgd104PgCsBd4GHAnsAJ4JzE1PT08lSeJuxS75bNiwwRZNqUVgAAl8976f8amb7uKnP9s337vXv+TpvOGlvzGAPVWXRKA4gZGRkWW+2GujvBC4E7gi69btwEnZ88rFnv4e8Azgw9k/fMud48C/N0OhZ5TFE8QpY/3FZ+t1OHU/8uvHZ5GtiPcjv/q2qn22cyVWfr02Svfc8VzgFOB4oJY9q6yPxvOAzwGvA54KXJONLOdklLakbaaONZHDkyhWY7/xa3wW6UaRf3Dqq3p93reE22/8Ghuq9hU7LxZVsfLr9QmzOOt1I3AvcAZwC+Am8OwEjs4Au9uvzlAfAn6/7lbtsqhpRFnNRLb1Opy6Xy4ErUaRl7z7pd9udmspHAFbTf3CTyNeWxwHjV+vjTJ4NGSUNqS6UMXPr92MVsU3/vi264HiW058ZZQ2rt5qJbI3siUC8WvNL8+zSPFT/tkI2NSx5p+M0hZ3b3WsieLd0ZIE4tcc7PL9ItPNzdZoFT9bYopfNfnJKG1x91brRPNGphFlB2RrJ6/aBOniwuVt34tU/in/bARs6ljzT0Zpi7u3OtZE8e5oSQLxOwA27yiyPhTiZ0tM8asmPxmlLe7eap1o3sg0omyCrGEUuQfS8dna2I2d6Cr/OhFq/734VZOfjNIWd2+1TjRvZDLKOgLLR5FcvHJ4eGrH5vUP5iGr/MtDqXUZ8asmPxmlLe7eap1o3shklBmBtedtPweYIuFQIPcoUrdebTknfuInowyXA7lqklHmwtSyUBX5jU5eecRjJJcDazIwXqNIXehtOSd+4iejDJcDuWqq4oU+F5icharGr2EUuTdJkvGZ6dPcxgCFPlXjVwhSG5H42YjGyk9GaYu7tzrWRPHuaEmCqvBziwc89vC+y0kYdShTuPag4eHxvM8iW+GvCr+S0k+bBhjBxpp/Mkpj4H3lsSaKbz/LKl8FfuvOv2o0nUudSbpnkeZRpG4dhsvGKuRfOFrLa4qVn4yyzKxoUnesidJlTC0PN8j8GkeRkN60EsZ31MbuDsV/kPmFYtSuHvGzUY6Vn4zSFndvdayJ4t3RkgSDyq9hIfO9acLUddOnbw2NcVD5heakW9flEI01/2SU5eRDJUdE3UAZ64nWik2zhcxXko6GHEXq1mu4zBy0/AtHJl9NsfKTUeaLb7BSsSZKMADGigaJX5El6Iz4NBnFCHCQ8s+IopA8Vn4yykLhLi6KNVGK9zisclD4+SxkHpLgoPALycSnLvHzobW8bKz8ZJS2uHurY00U746WJIidX8jFA4ogjp1fkT6H1IifjWas/GSUtrh7q2NNFO+OliSImV+IJeisWGPmZ+17CL342SjGyk9GaYu7tzrWRPHuaEmCGPmVtXhAEcQx8ivSz7I04mcjGys/GaUt7t7qWBPFu6MlCWLjV+biAUUQx8avSB/L1IifjW6s/GSUtrh7q2NNFO+OliSIhd/y1z7CLx5QBHEs/Ir0rRsa8bNRjpWfjNIWd291rIni3dGSBDHwO/uvvnnc0P45twTdaochTXhfGYsHFEEcA7+RkZG+vS6JX5GsO6CJlZ9vQrryqQ3VErWrbwswDtwPnAnc3FD/8cDX6/7tH4HXtWpDrVZLJycnffsVsEvtq4o1UboGqMOB+p3fX3zmy+n//uYPFnuxa25oaPz6C069VfzyEej3+Kp9+eLYqlSs/HwN5XvAFcA24B4bsnn1emAi+3sccClwTEO9vw28GfiDPMeTUeah1LpMrIls67Vd3ey1j9na6S63++qj+NrCIX7V5OdrlJ8GTgZ+BbgBcJvJXg08XBDfRcAdmfG6Ku76u5MAACAASURBVG7L6r+vrr7TgI8CTwZ+kRnm5zWiLEg88hFbP16o6ifsHHLwCh565LHXzNbGbiwnQrZa+5FffY/UPsXXRsCmbpV/vkbpWnEQ8CpgLfDO7P//F/AR4OeezbwMmAFmM50z37OA3XX1vARw/20H3KjzM8AzgMemp6enkiTZ1HjMDRs2eDZDxUXAn8BDj+zn7266i9vufmBe/MJnH8o71hyFM0t9REAE4iTQ7Bl5EaN8PvAW4K3u2gC4X87u77eA13uiuRC4M7ud66S3AydlzytbVfVt4I1A/ajz8bK69eoZgYbi+kWfj1+r3T7ELx+/VqXET/xsBGzqUCPKfwWOBe7Nbrt+HNgDPBP4N2CVZzPXAecCpwBu0k4tGzXWV/NymN/A1k3iORH4JHAUMNfsWDJKzwjIKL2BNa7TWr/bhy703jiXCMRP/GwEbOpQRnkd4MzR3S7dX9ckd6/p94C/9Gzm4qzXjZn5ngHcAhwO7ASOBo4E/gZ4KeA2sP397Plo00PJKD0jIKPMDWx+wk6aXLP42gekm2drY1P1FehCnxtn04LiJ342AjZ1KKO0taILahmlDbIuVM35NaywswfS8WYTdsRP+WcjYFMr/8rhV+QZpa0lJatllDbAOtGW8nMr7Dy6b9+WZOFdX/cS8bUHDQ+P79i8/sFmpMVP+WcjYFMr/8rhJ6O0cfVWK5G9kS0RdJOf21g5mZu7JoEjgL0J6cRMbcy9R9zy0832FSGp9hWhdkAjftXkJ6O0xd1brRPNG1lPjLLoxsqKbxzxLdpKxbcouQVdrPxklLa4e6tjTRTvjpYkKJvf/JZY+/ZdA6zJunCxzwo7ZbfPilXtsxEUv2ryk1Ha4u6t1onmjaxrI8rGLbEgHfVdYUfx7d/42loW94goRN9D1BHr+SGjDBF9jzpiTRSPLpZatCx+687bviVN5tcddlN2blo5vGq01YSddh0sq32hoKp9NpLiV01+Mkpb3L3VOtG8kZU6onQTdkJuiaX49ld8ba1ZrlZ8bURj5SejtMXdWx1ronh3tCRBSH7rJq8cT9NkC8n8yk9BtsQK2b4yEKp9NqriV01+Mkpb3L3VOtG8kQUfUc5P2Hl4n9tYedRVnpBuWzG8aqLIrdbG3ii+vY+vrQXt1YqvjW6s/GSUtrh7q2NNFO+OliSw8ivybqRPV6zt8zlWkbJqXxFqBzTiV01+Mkpb3L3VOtG8kQUbUa49b/s5JGzNKtxVv5i5rVW6kIpfKAK2enR9KYefjNLG1VutRPZGZjbKxlutgNe7kT4tVnx9aC0vK37iZyNgU7fKPxmljau3WhcCb2Qmo3T7RrKw44ebsLM3SZLxmenTdtha0Vqt+NrIip/42QjY1DJKG79gal0IbCh9+C1dhq74u5E+LfZpn0+9ocqqfTaS4ldNfhpR2uLurdaJ5o3Me0S5fBm65ftG2lqhEaX4lUXAVq+uL+Xwk1HauHqrlcjeyLyMsvFWa5Fl6CwtVHwt9OJdNNvW63Bq5Z+NpW692vgFUyuRbSjb8evFrdbG3ii+5cXXVnMYteJr4xgrP40obXH3VseaKN4dLUnQjF8vb7XKKMMGWueHjaf4lcNPRmnj6q1WInsja3vrtde3WmWUtniKn/iFJWCrTbdebfyCqWWUNpT1/PrhVqsu9LZ4ip/4hSVgq01GaeMXTC2jtKF0/P7ok7f92tLNlbs3q7VT6xXfToTafy9+4mcjYFPLKG38gql1IbCh/Pudt6aXzOx+cHEBgW7Pau3UesW3EyEZpY2Q+PWCX6+fUbrjbwHGgfuBM4GbW4B4ArAbOAuYbQWrVqulk5OTve5Xy1jqQlo8zfvxVqtuHRaPZzOlzg8bT/Erh1+vDWU9zO8q7/4eB1wKHNOiqx8DXp4Zq4zSlg9RGXk/zWrthF0Xqk6ENCKyERK/XvDrtVFeBNwBbMs6fxtwMnBfA4xjgT8Efg7s1IiyvFTptwt947ZYZ699zpNOfsXqXudtVD806hvbb/HViDzsuaz42nj26zPKy4CZOuO7Ibu16m6x1n8+lRnlH9cb5fT09FSSJJsa0WzYsMFGS+q+IHDTHT/k6q/eM9+W33jyEzh77XM45OAVfdE2NUIERGAwCYyMjCz7Id7rX+YXAncCV2TIbwdOyp5XLkbhbcATgcuBmkaU5SZnP/wibbctVj+0r10E1D5bfoqf+NkI2NT9OqJcB5wLnAIcnxmhe1ZZ//kSsKbh35yZfrEZEk3mKSdRbLXmV7tbrUP75y4nYXWzbbF0Ic3PsllJ8RM/GwGbOtb86/WIcnHW60bgXuAM4Bbg8GzkeHRDWDSitOVpR3UvE3nd5JXjaZpsyV792DU3NDR+/QWn3lrf6F62ryM8tKh3HkYakVsptdbr/LCx7dcRpa1XTdQaUdqQ9upEW3fe9i1pMj8DmoR024rhVRM7Nq9/sLE3vWpfXqpqX15SzcuJn/jZCNjUMkobv2BqXQiWohydvPKIx9LkmuxWqzPJjTO1scVn1su4i58tFcVP/GwEbOpY86/Xt15t1DWiDM6vm4ncsKD5nrmhodHGW60aUYYNcTfjW6Tlal8Ragc04lcOPxmljau3Wom8gGztedvPIWHrwv+lN60cXjXa7FarjNI7xdoKlH82nuJXTX4ySlvcvdVVP9GWv/rht6B51fl5J1yDQPxsBMWvmvxklLa4e6urfKJ1evUjD8wq88vDp1MZ8etEqP334ldNfjJKW9y91VU90dadf9VoOpe69yMPBXatJB3dURu72xdgVfn5cmpVXvxsJMWvmvxklLa4e6ureKLlffUjD8wq8svDJW8Z8ctLqnk58asmPxmlLe7e6iqdaI27fqQJ77tu+vRsAo83unlBlfgVI6Rbh2VwW6xT+WejGys/GaUt7t7qWBPFt6ONu36E2mC5Kvx8eectL355SWlEaSM1WPxklGVkQ5s6q3ChalyKbuXw8Jo8r37kCUUV+OXhULSM+BUlt6ATv2ryk1Ha4u6tHvQTrfF55ExtbNwbUsV/aITk1VjXoOdfmexklHa6seafjNIee68aYk2UTp1sfB7ZaSm6TvW1+n5Q+RXl4asTP19iS8uLXzX5yShtcfdWD+KJ1vg8cm5oaE2npei8wWWCQeRXlEURnfgVoXZAI37V5CejtMXdWz1oJ1qZzyObwR00ft4JZBSInw2g+FWTn4zSFndv9SCdaGsnr9oE6ZSD4LbGCv08UkbpnV4dBYOUfx07W0IB8bNBjZWfjNIWd291rIlS39HG9VpDvB+ZF+Qg8Mvb1zLKiZ+NqvhVk5+M0hZ3b3XsJ1rjeq2h3o/MCzJ2fnn7WVY58bORFb9q8pNR2uLurY75RGvYP7Lweq3e0OoEMfOz9DuUVvxsJMWvmvxklLa4e6tjPdHmJ+2QXO46nMK1Bw0Pj4daRMAHYqz8fPpYZlnxs9EVv2ryk1Ha4u6tjvFEe9Pk9ssTyBYO8Ns/0htQB0GM/EIzsNQnfhZ6WpnHRi9efjJKa+Q99TFdqLJJO18iYTWwNyGdmKmNXeHZ5aDFY+IXtOOBKhM/G0jxqyY/GaUt7t7qWE60bi4i4AMxFn4+fepmWfGz0Ra/avLrtVG6429h4bbe/cCZwM0NoXgq8DfAK4D7gPcD17YKV61WSycnJ3vdr5bZFMOJNvGJb6xfsslywEXNbadZvLdurP0OpY8h/0ZGRnT+Fgy44lsQXCZrxa/XCbkemADc3+OAS4FjGrrq/v1ZwBeAFwOfAo6SUdoSopX6r6+9Ob36q/fMf+0WEVgxvGqiF5N2WrVPFwJb3MVP/GwEbOpY86/XRnkRcAewLcN/G3ByNnJsjMgQ8Frgw8CrZJS2hG2m7qdJOzLK8PF1NcZ6oSqHhn+t4ufPrF4RK79eG+VlwAwwm8G8ATgL2N0kHA+7CSXAKPBV9/309PRUkiSbGstu2LDBFs2KqR96ZD+f+Id/47v3/YxDDl7B+hOfxcuOfUrFKKi7IiACIgDNbv332igvBO4EFmdS3g6clD2vbDaifBHwWeAFgDPOZR89o/RL9caVds59y/Of9KoTXtjrvGjZiVh/kfpFpbzS4mdjK37V5NfrC+I64FzgFOB4oJY9q6yPxsuBYeArwAiwEzgS+LmM0pa0mUm61z8OBXatHB5e86dnvOgBTaYozlUX0uLsdGvYxk78yuPXa6NcnPW6EbgXOAO4BTg8M8SjgecBbkUYN5p8APhINumnKRWNKPMlS6uVdnShz8evVSnxEz8bAZta+VcOv14bpa1XuvVaiF/99ljAxbO1093M4/mPTrRCSB8XiZ/42QjY1Mq/cvjJKG1cvdW9TuT6ma0J6cbGlXZ63b5OQNW+ToTafy9+4mcjYFPHmn8ySlvcvdW9SpTG5ehabY/Vq/blBan25SXVvJz4iZ+NgE0da/7JKG1x91b3IlEaZrbumRsaGr3+glNvbdb4XrTPB6La50NreVnxEz8bAZs61vyTUdri7q3udqI0m9nabqWdbrfPF6Da50tsaXnxEz8bAZs61vyTUdri7q3uZqLMz2xNky3u9Y+8e0h2s33e8DTZqAiyJRrF14ZQ/KrJT0Zpi7u3ulsnWv3rH27N1pnaWLafZPsmd6t93uAygdpXlNyCTvzEz0bApo41/2SUtrh7q7uRKPUzW9OE9103ffrWvA3tRvvytqVZObXPQk9GaaMnflXlJ6O0Rt5TX/aFvtPrH52aW3b7Oh2/0/dqXydCumNgIyR+4rd8mzcZZZlZ0aTusi70ja9/zA0NrWk1s7Vdl8tqXyjMap+NpPiJn42ATR1r/skobXH3VpeRKKFMUs+wvMO5TFBGfO2tOlCD2mejKX7V5CejtMXdWx36RPN9/aNTg0O3r9PxfL9X+3yJLS0vfuJnI2BTx5p/Mkpb3L3VIRMltElqROkdTo0o7ciW1BDy/AjctPnq1D4b1Vj5yShtcfdWh0qU+nckfV7/6NTgUO3rdJyi36t9Rckt6MRP/GwEbOpY809GaYu7tzpEohR9RzJPY0O0L89xipZR+4qSk1HayIlflfnJKENE36MO64V+7XnbzyEhey8y3TxbG5vyOHzHotb2dTyAsYDaZwMofuJnI2BTx5p/Mkpb3L3VlkSxviOZp7GW9uWp31pG7bMRFD/xsxGwqWPNPxmlLe7e6qKJ0g2TdJ0p2j5vEAUFal9BcJlM/MTPRsCmjjX/ZJS2uHurfRPFvSP56L59WxJwa7XuLbqQQN6G+rYvb72hyql9NpLiJ342AjZ1rPkno7TF3VvtkyghFxLI21Cf9uWtM2Q5tc9GU/zEz0bApo41/2SUtrh7q/MmSi9MUrdevcO5TJA3vvYjFatB7SvGbVElftXkJ6O0xd1bnedEazDJXStJR3fUxu72PlgBQZ72Fag2mETts6EUP/GzEbCpY80/GaUt7t7qTokyOnnlEY+lyTUkrAZ2rRweXrNj8/oHvQ9UUNCpfQWrDSZT+2woxU/8bARs6ljzr9dG6Y6/hYWJKvcDZwI3N4TiScAngdcCPwU+AHy6VbhqtVo6OTnZ6361zKZ2iVLGknS+aR1rIvv2s6zy4mcjK37iZyNgU7fKv14bynpgAnB/jwMuBY5p6OoLAfff54DnADuAIwfNKPvBJB1TXajKOdFstYZTK742luJXTX69NsqLgDuAbRn+24CTgftahONFwJ8Drxoko6w3yRSuPWh4eLybt1vrWepCUM0Lga3X4dTKPxtL8SuHX6+N8jJgBpjNuncDcBawu0l3nwv8JfC7buDjvp+enp5KkmRTY9kNGzbYaHVR/YOf/JJLZnfz0CP7OeHYp/COV7ccLHexVTqUCIiACFSTwMjIyDJf7LVRXgjcCVyRheR24KTseWV9lNYAk8DGNqPN+fIxPaOsH0mG3AHEkt76RWqhp1vXNnriJ35WAjZ9vz6jXAecC5wCHO98LntWWd/bZwN/DawFHuqEIRaj7EeTdGxllJ0yrP334id+NgI2tfKvHH69HlEuznp1I8V7gTOAW4DDgZ3A0cCHgY80dP/XgKavTMRglGf/1TePG9o/9yUSDu2XkeQiX51o5ZxotlrDqRVfG0vxqya/XhuljXoTdb8b5Y5//Eb6iX/47oP9aJIaUdrTURdSG0PxEz8bAZu6X2+92noVmVGWueFyKJC6UNlIip/42QjY1Mq/cvhpRGnjmlsdg0lqRJk7nC0L6kJlYyh+4mcjYFNrRGnjZ1LPT9yZm/u2q8S9AvJfz3xd3/5A0YXKFGpNhrLhEz/xMxKwyWWUNn6F1Y2zW7e+54R3NXtPp/ABAgtllDag4id+NgI2tfKvHH59O7Ip2t1+mszT7BUQJXLRyC7oxE/8bARsauVfNfnJKG1xb6lu9Z6kTjQbcPETPxsBm1r5V01+Mkpb3Juq2y0moBPNBlz8xM9GwKZW/lWTn4zSFvdl6k4r7uhEswEXP/GzEbCplX/V5CejtMV9ibqTSeoZmx22LlQ2huInfjYCNnWs+SejtMX9cfXo5JVHPJYm33Yr7ritsq6rnT7arOpYEyUQJnM14mdDKH7iZyNgU8eafzJKW9zn1aObrjn0sYf3ubVbVwO7Vg4Pr2m1n2SsiRIAU5AqxM+GUfzEz0bApo41/2SUtrh7maRuvRph6/UQM8BYL1TmjgeqQPxsIGPlJ6M0xN1nJLl4mFgTxYApqFT8bDjFT/xsBGzqWPNPRmmI+9rJ7V8C3KbSe1YOD69udbu1/hCxJooBU1Cp+Nlwip/42QjY1LHmn4yyYNzfNLn98gTGgb1zQ0Nrrr/g1FvzVBVrouTpWzfKiJ+NsviJn42ATR1r/skoC8S9qEnqGWUB2A2SWE80e8/D1CB+No7iV01+MkrPuNdvlwXpa2ZrYzf6VKETzYfW8rLiJ342Aja18q+a/GSUHnFv2FNy40xt7AoP+XxRnWi+xJaWFz/xsxGwqZV/1eQno8wZ9/o9JRPSQiYpo8wJu00xXahsDMVP/GwEbOpY809GmSPueZamy1GNRpR5IckoA5BqXkWsF6rSgHhWLH6ewBqKx8pPRtkh7vXvSrZbmi5v+sSaKHn7V3Y58bMRFj/xsxGwqWPNPxllh7ivPW+7W7+149J0edMn1kTJ27+yy4mfjbD4iZ+NgE0da/712ijd8bew8D7i/cCZwM1NQnEisA34M+DSdqGq1Wrp5ORkkH7Vvwaycnj4iDwLCnRKo1gTpVO/uvW9+NlIi5/42QjY1LHmXxBDMaBbD0wA7u9xmQke01DfwZmZPgB8v1tGuXbyyilINvkuKNCJRayJ0qlf3fpe/GykxU/8bARs6ljzr9dGeRFwRzZadBG4DTgZuK9JOD4E/LgbRrnu/KtG0zS9xrUhSZL1M9On7bClxwF1rIkSqv/WesTPRlD8xM9GwKaONf96bZSXATPAbIb/BuAsYHceo5yenp5KkvlR35LPhg0bCkfzBz/5JZfM7uahR/az/sRnseYFTy1cl4QiIAIiIAJxERgZGVnmi702yguBO4HFF/dvB07Knlc20i19ROlmuD66b9+3EzgiId02Uxtzz06DfmL9RRUUgqEy8TPA04IXNnjiV1l+vTbKdcC5wCnA8UAte1bZLCClG2XdbiBtN1+2ZIsu9BZ6WtnIRk/8xM9KwKaP9frXa6NcnPW6EbgXOAO4BTgc2AkcDbwC+HJDeA7Lnlcui1rRWa/1k3dCzXBtllKxJort9AinFj8bS/ETPxsBmzrW/Ou1UdqoN1EXMcr6yTtFFjr36USsieLTxzLLip+NrviJn42ATR1r/lXeKEcnrzzisTRxiwocmia877rp07faUqG9OtZEKZOJT93i50NreVnxEz8bAZs61vyrvFEurrwTYnm6PCkUa6Lk6Vs3yoifjbL4iZ+NgE0da/5V2ijXTm53o8dzgD0rh4dXh1h5p1MaxZoonfrVre/Fz0Za/MTPRsCmjjX/KmuUdc8l984NDa25/oJTb7WlQD51rImSr3fllxI/G2PxEz8bAZs61vyrpFFmO4Lc1a3nkvWpFWui2E6PcGrxs7EUP/GzEbCpY82/Shrlgfcl05tma2NrbKH3U8eaKH69LK+0+NnYip/42QjY1LHmX+WM8k3nb59I0vkdS/aW+b5kq3SKNVFsp0c4tfjZWIqf+NkI2NSx5l+ljPKNH/z06qH9c19yt1xDL3aeN31iTZS8/Su7nPjZCIuf+NkI2NSx5l+ljLJuE+aLZ2unu+29uv6JNVG6DqrFAcXPFgnxEz8bAZs61vyrjFHWLVHXtVdBmqVUrIliOz3CqcXPxlL8xM9GwKaONf8qYZT1q++UvURdpzSKNVE69atb34ufjbT4iZ+NgE0da/5VwijXnrf9GhJGu7X6TrtUijVRbKdHOLX42ViKn/jZCNjUsebfwBtl/cICvZjl2phWsSaK7fQIpxY/G0vxEz8bAZs61vwbaKOs34i5Gwue50mhWBMlT9+6UUb8bJTFT/xsBGzqWPNvoI2ybi3XXbO101fbQhxGHWuihOm9vRbxszEUP/GzEbCpY82/gTXK+Qk8JHe5sM4NDR3XrbVcO6VRrInSqV/d+l78bKTFT/xsBGzqWPNvYI1y8XWQhHTbTG1s3BbecOpYEyUcAVtN4id+NgI2tfKvmvwG1yjP2/6AW4Gn16+DNKaVTrRqnmi2XodTK/9sLMWvmvwG0ii/8uCzN6YklwN982xyMb10olXzRLP1Opxa+WdjKX7V5DeQRrnzwSNuBNYkpBtnamNX2EIbVq0TzcZT/MTPRsCmVv5Vk9+gGqWLZk92B+mURjrROhFq/734iZ+NgE2t/Ksmv0E2yp4tfN4ulXSiVfNEs/U6nFr5Z2MpftXk12ujdMd3e0O6Wan3A2cCNzeEIk+ZeYlbYODlh/zrAzsfPIKVpEfuqI3dbQtreLVONBtT8RM/GwGbWvlXTX69Nsr1gNvuyv09DrgUOKYhFHnKzEvcpsyvfNLdW7784BHXXlc7fdQW0nLUOtFsXMVP/GwEbGrlXzX59dooLwLuALZl+G8DTgbuqwtHnjILRjm5fccrD737lK/sPXL9zPRpO2whLUetE83GVfzEz0bAplb+VZNfr43yMmAGmM3w3wCcBeyuC0fLMtPT01NJkmyqD92KlSvZ/9hjtmhKLQIiIAIiUDkChx12GO9+97uX+WKvjfJC4E5g8RWO24GTsueVi0HKU+bxgNZqtXRycrLX/WqZYGqf7dwTP/GzEbCplX/V5NdrQ1kHnAucAhwP1LJnlfXRyFNGRmnLX/ETv0AEbNXIiMTPRsCmbpV/vTbKxRmtG4F7gTOAW4DDgZ3A0UCrMk2J6EQrJ1FstYZTK742luInfjYCNnWs+ddro7RRb6KONRDBQRSsUPwKgstk4id+NgI2tfKvHH4DZ5Rugs/5558/ZcNVnlrts7EVP/GzEbCplX/V5DdwRmkLo9QiIAIiIAIisJSAjFIZIQIiIAIiIAJtCMgolR4iIAIiIAIiUEGjzLM+bJ4yRZMnb90nZqsS/Vm2fF/R4/nq8rTvScAngdcCPwU+AHza90AFy+dp31OBvwFeka3k9H7g2oLH85Xlad9inU/IFtBwC2ksLqzhezzf8nna517H+npdxf8IvM73QAXL52mfq/pdwMcAx9AtLHJJweP5yvK070PARxsqPtTtWuR7sALl87TvIMDtyftm4EfZUqFucZdufPK0bwXwceCtwD3A7wC3Bm5cp+trnnbON2lQR5R51ofNU6Zo3PLUfXC2IPwDwPe7bJR52vdCwP33OeA5gFsS8MiiQDx1edrn1gZ+FvAF4MXAp4CjPI9TtHie9i3W7S70L89i3S2jzNO+384uon9QFIJBl6d97hUxF1v3HvUvsth+zXBMH2me9tXX586PP8neB/c5TtGyedr3tsx83gEcC3y2z86PU7MfQq59bwDOBn6rKJAmujzX1zwcB9oo86wPm6dM0bj51O1+mf64y0bp0z7H4EXAnwOvKgrEU+fTvqFs1PvhPmyfu0D9IfDz7L3gbhllHn6nZSOiJ2dG5Azz855xKlo8T/ve6ZZvzn5krMoupO5i341PnvbVt8MZ+nuBu7rROCBP+14DuLssb8/eR3dLgZ7QR+1zdwh+AHwia9MPgacD+wO3sd31NQ/HgTZK0xqyAQKV5/iLh+mFUfq077nAXwK/C3wvAJs8Vfi07+HsdpfbLeareSoPUCZv+9wo1xnlH3fZKPO07yWA+297thrWZ4BnAN1YKDlP+9yt/rWAGxm5OxnujsYzgbkA8etURZ72Ldbh2uhuWbtdkLr1ydM+d7fQMXO3XvcBbwGu71ID87TPGbgbTY4B7hapuy3868CDgdvY7vqap50DbZR51ofNU6ZozHzq7oVR5m3fGmAScCsn1e/oUpRLXl3e9rn63IjSjXjdaOMFgDPOsj952ucu8E/MnhO5pRndSlPdGlHmaV8jo28Db+xSnPO07/cy43Z3CtznW8DrgX8vO7hAnvYtNsOZjxu51W/kUHYT87TvPdkPIPeD42mZSbpHFP1yfrhnqO6HpMu5q7O5BmU8Oml3fc3DcaCNMs/6sHnKFE14n7p7YZR52vds4K+zX/UPFQVRUJenfe653zDwFWAkMyI38nC3Ocv+5GnflwD3Q6P+4xb8/2LZjcue63VaQ9nxc5NP3CQe94veTdxyF6pujNjy8Hte9nzcjdbcxK1rspFlv7TPhdFNMtrVZA/dskOch5+7rrhbmc4o3fPem7N2/qzsxuXMP3f34qXZ+eBGl+45pRv1hv60u77m4TjQRhl8DVnP6OU5vput+eWGeg/Lnld6Hs67eJ72uV/yH2mo+ddKuDXSrPF52ucupG5WnxtNuglRrq1u4+9ufPK0r74d3R5R5mmf+1HhZg27i9XdwO8Dbpu7bnzytM+1w13kneG7H2qufd0akedtn3ss4XLu1d2AVneMPO17CvC3wCuzRxMXFRdFSwAAAdlJREFUZPMMutHUPO371eyZ+MuAb2S3Yd3s11CfVtfXlUXWER/UWa+hYKseERABERCBihOQUVY8AdR9ERABERCB9gRklMoQERABERABEWhDQEap9BABERABERABGaVyQAREQAREQASKEdCIshg3qURABERABCpCQEZZkUCrmyIgAiIgAsUIyCiLcZNKBERABESgIgRklBUJtLopAiIgAiJQjICMshg3qURABERABCpCQEZZkUCrm5Un8F+B/5ItWed2aPjnbONct0ycPiIgAm0IyCiVHiJQDQJujct/yrr6k2xBdLce5qPV6L56KQLFCcgoi7OTUgRiI+AWknfbVa3Idr3v1kbDsXFSe0VgCQEZpRJCBKpDwO208nXgYOC9dbvLV4eAeioCBQjIKAtAk0QEIiSwCrgFuB34YbYZ9/OBH0TYFzVZBLpKQEbZVdw6mAj0jMDF2Z5/bg/FXwJ3ZP+5zWv1EQERaENARqn0EAEREAEREAEZpXJABERABERABIoR0IiyGDepREAEREAEKkJARlmRQKubIiACIiACxQjIKItxk0oEREAERKAiBGSUFQm0uikCIiACIlCMgIyyGDepREAEREAEKkJARlmRQKubIiACIiACxQj8f1fZJzKUyb8GAAAAAElFTkSuQmCC",
      "text/plain": [
       "<VegaLite 2 object>\n",
       "\n",
       "If you see this message, it means the renderer has not been properly enabled\n",
       "for the frontend that you are using. For more information, see\n",
       "https://altair-viz.github.io/user_guide/troubleshooting.html\n"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import altair as alt\n",
    "import numpy as np\n",
    "\n",
    "data = pd.DataFrame(zip(np.arange(0,1,0.01), np.power(np.arange(0,1,0.01), 0.75)), columns=[\"x\", \"y\"])\n",
    "alt.Chart(data, title=\"x^0.75\").mark_line().encode(x=\"x\", y=\"y\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "class NoiseDistribution:\n",
    "    def __init__(self, vocab):\n",
    "        self.probs = np.array([vocab.freqs[w] for w in vocab.words])\n",
    "        self.probs = np.power(self.probs, 0.75)\n",
    "        self.probs /= np.sum(self.probs)\n",
    "    def sample(self, n):\n",
    "        \"Returns the indices of n words randomly sampled from the vocabulary.\"\n",
    "        return np.random.choice(a=self.probs.shape[0], size=n, p=self.probs)\n",
    "        \n",
    "noise = NoiseDistribution(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this distribution, we advance through the documents creating examples. Note that we are always putting the positive sample first in the samples vector, following the convention the loss function expects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "def example_generator(df, context_size, noise, n_negative_samples, vocab):\n",
    "    for doc_id, doc in df.iterrows():\n",
    "        for i in range(context_size, len(doc.clean_tokens) - context_size):\n",
    "            positive_sample = vocab.word2idx[doc.clean_tokens[i]]\n",
    "            sample_ids = noise.sample(n_negative_samples).tolist()\n",
    "            # Fix a wee bug - ensure negative samples don't accidentally include the positive\n",
    "            sample_ids = [sample_id if sample_id != positive_sample else -1 for sample_id in sample_ids]\n",
    "            sample_ids.insert(0, positive_sample)                \n",
    "            context = doc.clean_tokens[i - context_size:i] + doc.clean_tokens[i + 1:i + context_size + 1]\n",
    "            context_ids = [vocab.word2idx[w] for w in context]\n",
    "            yield {\"doc_ids\": torch.tensor(doc_id),  # we use plural here because it will be batched\n",
    "                   \"sample_ids\": torch.tensor(sample_ids), \n",
    "                   \"context_ids\": torch.tensor(context_ids)}\n",
    "            \n",
    "examples = example_generator(example_df, context_size=5, noise=noise, n_negative_samples=5, vocab=vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we package this up as a good old PyTorch dataset and dataloader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset, DataLoader\n",
    "\n",
    "class NCEDataset(Dataset):\n",
    "    def __init__(self, examples):\n",
    "        self.examples = list(examples)  # just naively evaluate the whole damn thing - suboptimal!\n",
    "    def __len__(self):\n",
    "        return len(self.examples)\n",
    "    def __getitem__(self, index):\n",
    "        return self.examples[index]\n",
    "    \n",
    "dataset = NCEDataset(examples)\n",
    "dataloader = DataLoader(dataset, batch_size=2, drop_last=True, shuffle=True)  # TODO bigger batch size when not dummy data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's going to also be useful to have a way to convert batches back to a readable form for debugging, so we add a helper function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'doc_id': tensor(2),\n",
       "  'context': 'was allowed a moment to ____ in at him where he',\n",
       "  'context_ids': tensor([ 99,   5,   0,  61,  93,  52,  11,  48, 103,  47]),\n",
       "  'samples': ['peer', 'moment', 'atreides', 'a', 'caladan', 'night'],\n",
       "  'sample_ids': tensor([71, 61, 12,  0, 20, 65])},\n",
       " {'doc_id': tensor(3),\n",
       "  'context': 'mother the old woman was ____ witch shadow hair like matted',\n",
       "  'context_ids': tensor([ 62,  91,  67, 105,  99, 104,  79,  44,  59,  60]),\n",
       "  'samples': ['a', 'where', 'in', 'cooled', 'an', 'woman'],\n",
       "  'sample_ids': tensor([  0, 103,  52,  24,   6,  -1])}]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def describe_batch(batch, vocab):\n",
    "    results = []\n",
    "    for doc_id, context_ids, sample_ids in zip(batch[\"doc_ids\"], batch[\"context_ids\"], batch[\"sample_ids\"]):\n",
    "        context = [vocab.words[i] for i in context_ids]\n",
    "        context.insert(len(context_ids) // 2, \"____\")\n",
    "        samples = [vocab.words[i] for i in sample_ids]\n",
    "        result = {\"doc_id\": doc_id,\n",
    "                  \"context\": \" \".join(context), \n",
    "                  \"context_ids\": context_ids, \n",
    "                  \"samples\": samples, \n",
    "                  \"sample_ids\": sample_ids}\n",
    "        results.append(result)\n",
    "    return results\n",
    "\n",
    "describe_batch(next(iter(dataloader)), vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's jump into creating the model itself. There isn't much to it - we multiply the input paragraph and word matrices by the output layer. Combining the paragraph and word matrices is done by summing here, but it could also be done by concatenating the inputs. The original paper actually found concatenation works better, perhaps because summing loses word order information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class DistributedMemory(nn.Module):\n",
    "    def __init__(self, vec_dim, n_docs, n_words):\n",
    "        super(DistributedMemory, self).__init__()\n",
    "        self.paragraph_matrix = nn.Parameter(torch.randn(n_docs, vec_dim))\n",
    "        self.word_matrix = nn.Parameter(torch.randn(n_words, vec_dim))\n",
    "        self.outputs = nn.Parameter(torch.zeros(vec_dim, n_words))\n",
    "    \n",
    "    def forward(self, doc_ids, context_ids, sample_ids):\n",
    "                                                                               # first add doc ids to context word ids to make the inputs\n",
    "        inputs = torch.add(self.paragraph_matrix[doc_ids,:],                   # (batch_size, vec_dim)\n",
    "                           torch.sum(self.word_matrix[context_ids,:], dim=1))  # (batch_size, 2x context, vec_dim) -> sum to (batch_size, vec_dim)\n",
    "                                                                               #\n",
    "                                                                               # select the subset of the output layer for the NCE test\n",
    "        outputs = self.outputs[:,sample_ids]                                   # (vec_dim, batch_size, n_negative_samples + 1)\n",
    "                                                                               #\n",
    "        return torch.bmm(inputs.unsqueeze(dim=1),                              # then multiply with some munging to make the tensor shapes line up \n",
    "                         outputs.permute(1, 0, 2)).squeeze()                   # -> (batch_size, n_negative_samples + 1)\n",
    "\n",
    "model = DistributedMemory(vec_dim=50,\n",
    "                          n_docs=len(example_df),\n",
    "                          n_words=len(vocab.words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take it for a spin!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0., 0., 0.]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    logits = model.forward(**next(iter(dataloader)))\n",
    "logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oh yeah, the output layer was initialized with zeros. Time to bash out a standard issue PyTorch training loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm, trange\n",
    "from torch.optim import Adam  # ilenic uses Adam, but gensim uses plain SGD\n",
    "import numpy as np\n",
    "\n",
    "def train(model, dataloader, epochs=40, lr=1e-3):\n",
    "    optimizer = Adam(model.parameters(), lr=lr)\n",
    "    training_losses = []\n",
    "    try:\n",
    "        for epoch in trange(epochs, desc=\"Epochs\"):\n",
    "            epoch_losses = []\n",
    "            for batch in dataloader:\n",
    "                model.zero_grad()\n",
    "                logits = model.forward(**batch)\n",
    "                batch_loss = loss(logits)\n",
    "                epoch_losses.append(batch_loss.item())\n",
    "                batch_loss.backward()\n",
    "                optimizer.step()\n",
    "            training_losses.append(np.mean(epoch_losses))\n",
    "    except KeyboardInterrupt:\n",
    "        print(f\"Interrupted on epoch {epoch}!\")\n",
    "    finally:\n",
    "        return training_losses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll sanity check by overfitting the example data. Training loss should drop from untrained loss to something close to the minimum possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epochs: 100%|██████████| 40/40 [00:02<00:00, 21.83it/s]\n"
     ]
    }
   ],
   "source": [
    "training_losses = train(model, dataloader, epochs=40, lr=1e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.vegalite.v2+json": {
       "$schema": "https://vega.github.io/schema/vega-lite/v2.6.0.json",
       "config": {
        "view": {
         "height": 300,
         "width": 400
        }
       },
       "data": {
        "name": "data-b4211a284b320247988d74ee339a35c7"
       },
       "datasets": {
        "data-b4211a284b320247988d74ee339a35c7": [
         {
          "epoch": 0,
          "training_loss": 3.873234655897496
         },
         {
          "epoch": 1,
          "training_loss": 2.4964269541077693
         },
         {
          "epoch": 2,
          "training_loss": 1.8195302142935283
         },
         {
          "epoch": 3,
          "training_loss": 1.41878628124625
         },
         {
          "epoch": 4,
          "training_loss": 1.1586913345223766
         },
         {
          "epoch": 5,
          "training_loss": 0.9543892801818201
         },
         {
          "epoch": 6,
          "training_loss": 0.8089911760920185
         },
         {
          "epoch": 7,
          "training_loss": 0.690002828836441
         },
         {
          "epoch": 8,
          "training_loss": 0.6072199561838376
         },
         {
          "epoch": 9,
          "training_loss": 0.5273456401744131
         },
         {
          "epoch": 10,
          "training_loss": 0.4652211883310544
         },
         {
          "epoch": 11,
          "training_loss": 0.4105674152152013
         },
         {
          "epoch": 12,
          "training_loss": 0.36498250900688817
         },
         {
          "epoch": 13,
          "training_loss": 0.3294624433679096
         },
         {
          "epoch": 14,
          "training_loss": 0.29689174738980956
         },
         {
          "epoch": 15,
          "training_loss": 0.2668858468532562
         },
         {
          "epoch": 16,
          "training_loss": 0.24272207208609176
         },
         {
          "epoch": 17,
          "training_loss": 0.21813622003389618
         },
         {
          "epoch": 18,
          "training_loss": 0.1999257556715254
         },
         {
          "epoch": 19,
          "training_loss": 0.18043559644434412
         },
         {
          "epoch": 20,
          "training_loss": 0.16454930458281
         },
         {
          "epoch": 21,
          "training_loss": 0.14988945260391398
         },
         {
          "epoch": 22,
          "training_loss": 0.1378944904496104
         },
         {
          "epoch": 23,
          "training_loss": 0.12689736037183616
         },
         {
          "epoch": 24,
          "training_loss": 0.11709884914048647
         },
         {
          "epoch": 25,
          "training_loss": 0.10717140561190702
         },
         {
          "epoch": 26,
          "training_loss": 0.09873407258320663
         },
         {
          "epoch": 27,
          "training_loss": 0.09167827893111666
         },
         {
          "epoch": 28,
          "training_loss": 0.08547388257111534
         },
         {
          "epoch": 29,
          "training_loss": 0.0792784820610689
         },
         {
          "epoch": 30,
          "training_loss": 0.07406510791536104
         },
         {
          "epoch": 31,
          "training_loss": 0.06805220107405872
         },
         {
          "epoch": 32,
          "training_loss": 0.064012377917514
         },
         {
          "epoch": 33,
          "training_loss": 0.06023278630385965
         },
         {
          "epoch": 34,
          "training_loss": 0.05617725763911918
         },
         {
          "epoch": 35,
          "training_loss": 0.05255285915681871
         },
         {
          "epoch": 36,
          "training_loss": 0.04945897159434981
         },
         {
          "epoch": 37,
          "training_loss": 0.047309281130842235
         },
         {
          "epoch": 38,
          "training_loss": 0.04321649504857043
         },
         {
          "epoch": 39,
          "training_loss": 0.040696044799761244
         }
        ]
       },
       "encoding": {
        "x": {
         "field": "epoch",
         "type": "quantitative"
        },
        "y": {
         "field": "training_loss",
         "scale": {
          "type": "log"
         },
         "type": "quantitative"
        }
       },
       "mark": "bar"
      },
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcoAAAFfCAYAAADON4wsAAAgAElEQVR4Xu2dDbRdZXnn/yfBAHZKpKyMKVBFr3wo8YMgo3QxDK3aJOXix1qkXBjkS6iDtdVlLbnpkkliV3tvtCPt1FGsjBJQuDBxquYyCeMMuvADK0OsCC5Q6TAzELLGjwhjJUTCnfXk7nOzz8k5Z3++Z+9nn99eiwWc+348+/8+7/Pfz/O+7/O2xAMCIAACIAACINAXgRbYgAAIgAAIgAAI9EcAokQ7QAAEQAAEQGAAAhAl6gECIAACIAACECU6AAIgAAIgAAL5EMCjzIcbtUAABEAABEYEAYhyRAaa1wQBEAABEMiHAESZDzdqgQAIgAAIjAgCdSPKMyVtkfQRSddHY2AyXifpMkm7JV0h6RsjMj68JgiAAAiAQMUI1Ikol0SEuEfSYzGifJuk90qyf58W/X5ixbjRPQiAAAiAwIggUCeibEP+AUk/jhHlhyU9EHmaVuZ+SaskPTEiY8RrggAIgAAIVIiAB6L8hKRtkmYjnO6S9C5JD01NTW1stVob4vgtW7ZM55xzToWQ0jUIgAAIgIBXBMbGxg7hRQ9E+SFJ35N0YwT8dyW9KVqvPGQspqen5yYnJ+v4Xj315pFHHpnrNTB1VjJvMnuT18bem8ze5AXj4VgYb3rRT946Ekp36PU8SddIeoukMyRNR2uVPUcaogw/AZqi/OGRyt8DGOfHLm1NME6LVP5yTcG4TkR5lqSvdg3JMkk/iTb5XC5pl6RLJN3bb+ggyvxKnbZmU5Q/7ftWUQ6Mw6MOxmDcjYAnj7LQ6EGUheBLVRkDkwqmQoXAuBB8qSqDcSqYChVqCsZ18igzD0ivzTzWyNq1azO3RQUQAAEQAAEQ8LKZp9BI4VEWgi9V5aZ8JaZ62YoKgXF44MEYjAm9hteBUnrwNlntpb3J7E1eMC5laiU24k0vvMnbJD12HXrtNRPwKBPtQ+EC3iasN3mbZGAKK1vABrzphTd5m6THEGXAiZimaZQ/DUrFyoBxMfzS1AbjNCgVKwPGxfBLU7uRu17ZzJNm6CkDAiAAAiCQFgE286RFaojl+EoMDzYYg3EvBLzphTd5Cb2Gn3e5e2CNMjd0qSt6m7De5G2SgUmtVBUU9KYX3uRtkh6zRlnBBI13ifKHHwAwBmM8yvA60GSMIcpq9GehV4x4+AEAYzBushEPP7r5e/A299jMk3+sqQkCIAACIDAiCLCZp4YD7e2Lq0nrDjVUB7eRBvQ4vDaBcXUYE3oNj/3AHlD+8AMAxmBM6DW8DjQZY4iyGv1x6zngUQ5HYbyRuzd50WP0OAuxQ5TD0Ze+vWBgwg8AGINxFqMYHq18PaDH+XDLUquRm3l6AcA5yixqka+stwnrTV68nXx6mbWWN73wJm+T9BiPMuvsKrk8yl8yoD2aA2MwxqMMrwNNxtg1UZLrtRrlp1cQAAEQaCoCHA+p4cji7YQfFDAG4yZ7O+FHN38P3uYea5T5xzpoTW+K1KR1h6ADW7Bxb3rhTV70uKCCpqzuTS8gypQDO+xi3hQJAzMcDfGmF97kRY/R4yxRBtdrlL1elF2v4SeAN6PoTV6MeHgdBmMwhignJ918AGDEw09YMAbjLEYxPFr5ekCP8+GWpRah1yxoDbEsyh8ebDAGY4gyvA40GWM3nlfaYSb0mhap/OW8EY83eQkL5tfNLDW96YU3eZukx66JknOUWcwCZUEABEAABJIQ4BxlEkIV/J2vxPCggzEYNzksGH508/fgbe6xRpl/rIPW9KZITQqnBB3Ygo170wtv8qLHBRU0ZXVvegFRphzYYRfzpkgYmOFoiDe98CYveoweZ4kyuF6j7PWibOYJPwG8GUVv8mLEw+swGIMxRMk5yqCzwBvxeJMXIx5UfRca96YX3uRtkh7jUQ5nTvbtBeUPPwBgDMZZvIfwaOXrAT3Oh1uWWqxRZkFriGVR/vBggzEYQ5ThdaDJGONRVqM/bsM/TQqnVDz0A7v3Ru7e5EWPh6P93vQCj3I4epG5F2+KhIHJPMS5KnjTC2/yose51DJzJW960UiiJDNPZr2lAgiAAAiAwAAEyMxTE/VYs+EzRy1+5rCvmDjHH/P80x77yS++bf89Oz2xsiYiEhaseCCa8iVeMYzoccUD0BQ9Zo2yAkWKiPLJrq6fmp2eWFqBOJm7bIryZ37xIVYA4/BggzEYdyPQyNBrr2H2kHAAogw/QeM9eDOIrJ8NRz+86YU3eZukx3iUw5mTHb1AlMMFHQMTHm8wBuNeCHjTCzzK8HqcugeIMjVUpRT0Nlmb9CVeygAGasSbXniTt0l6jEcZaBIOahaiHC7oGJjweIMxGONRhteB0npgjbI0KPs25M0oepO3SV/i4bUxfw/e9MKbvE3SYzzK/PMsd008ytzQ5aqIgckFW6ZKYJwJrlyFwTgXbJkqeV6jfKGkmyWdJekJSe+T9IV+b49HmUkvchX2NmG9ydukL/FcCjakSt70wpu8TdJjDx7laZJeJOlLkl4t6bOSXgpRDsma9OjG24T1Jm+TDEx1Wprcsze98CZvk/TYA1G2NX6RpDdIulbS2RBlsiEIVcLbhPUmb5MMTCgdLKNdb3rhTd4m6bEnotwrybLZvFXSPTYIXnO97t23X+u27OyY60csWazNl7rIYFeGjaINEAABEKglAt5zvZpH+SpJWyWtkGTEecjDGmV43fP2ZetN3iZ9iYfXxvw9eNMLb/I2SY89eJSvl3S4pK9LGpP0NUkvkfRziDK/kShS09uE9SZvkwxMET0LXdebXniTt0l67IEoXyHp05E3uUfSByVdzxplaDPSv31vE9abvE0yMNVpaXLP3vTCm7xN0mMPRJms8bEShF4zwZWrsLcJ603eJhmYXAo2pEre9MKbvE3SY4hySJMy3g0JB4YLOgYmPN5gDMa9EPCmF54TDmTSQDzKTHDlKtwU5c/18kOqBMbhgQZjMO5GAKIMrxOpe8CjTA1VKQW9GcQmhaxKGcBAjXjTC2/yNkmPCb0GmoSDmk1DlOetu+VULVp8XXc726Yu+J0KRO7o0tuE9SZvkwxM1bo6qH9veuFN3ibpMURZwUxOS5RzrUUPdIn34Oz0hJ0hrfTxNmG9ydskA1OpoiZ07k0vvMnbJD2GKCuYyRDlcEHHwITHG4zBuBcC3vSCNcrwepy6B4gyNVSlFPQ2WZv0JV7KAAZqxJteeJO3SXqMRxloEpaxRknotZzBwcCUg2OT1vuaZMTDj27+HrzNPTzK/GNdek08ytIhHdigt8mKER+OfnjTC2/yNkmPPXiUSyXdFF2x9VNJ75d0e7+p1JRzlLbrFY+yHIOJgSkHRzzK8DiCcT0x9kCUr5Rk/3xO0imSPh8lRe+JKEQZXtG8EY83eZv0JR5eG/P34E0vvMnbJD32QJTxmWDXbH10FC5uxqPMbwC7a2JgysOyX0tgDMa9EPCmF01Yo3y5pI9JulLSI4Rew0/MphhFb5O1SV/i1Wlpcs/e9MKbvE3SYy8e5TmSJiVdLumJ9hSYmpra2Gq1NnRPibVr1ybPkgpL7N23X+u27OyQ4Igli7X50pULv+3e87SmtnbmG1h+9JFaf37l+QYqRI6uQQAEQCAsAmNjY4fwogeifLGkT0kal/R0EkSsUSYhVPzv3r5svcnbpC/x4toWrgVveuFN3ibpsQeivDa6rDk+Y46W9LNeUwiiDGdY2i17m7De5G2SgQmvjfl78KYX3uRtkh57IMpMMwGizARXrsLeJqw3eZtkYHIp2JAqedMLb/I2SY8hyiFNyng3ZSUcOHfdbatarbntHa/Q0n+dnZpYHfK1vE1Yb/I2ycCE1MOibXvTC2/yNkmPIcqisy1H/ZKJckcXUd4JUXYOCgYmh5JmrALGGQHLURyMc4CWsUoTjoekeuVRCr1GHiVEmaAZGJhUU6dQITAuBF+qymCcCqZChSDKQvCVWxmPslw8k1rDwCQhVPzvYFwcw6QWwDgJoeJ/hyiLY1haCxBlaVCmaggDkwqmQoXAuBB8qSqDcSqYChWCKAvBV25liLJcPJNaw8AkIVT872BcHMOkFsA4CaHif4coi2NYWgsQZWlQpmoIA5MKpkKFwLgQfKkqg3EqmAoVgigLwVduZYiyXDyTWsPAJCFU/O9gXBzDpBbAOAmh4n+HKItjWFoLEGVpUKZqCAOTCqZChcC4EHypKoNxKpgKFYIoC8FXbmWIslw8k1rDwCQhVPzvYFwcw6QWwDgJoeJ/hyiLY1haC8MkyjdP3npaL8G/OH3ht/O+kLcJ601eGxdvMnuTF4zzzv5s9bzpBUSZbXyDlh4mUY5Pztj9nZ/seqEbZqcnrsr7kk1R/rzvP4x6YBweZTAG424EIMrwOpG6B4gyNVSlFPRmEPF2Shn2xEa86YU3eZukx+R6TZxO5ReAKMvHdFCLGJjweIMxGPdCwJte4FGG1+PUPUCUqaEqpaC3ydqkL/FSBjBQI970wpu8TdJjPMpAk3BQsxDlcEHHwITHG4zBGI8yvA6U1gO3h6jjmi028/jbQdqkL/HSJnaAhryRuzd5m6THeJQBJmBSk3iUSQiV+3cMTLl4NsFzaJIRDz+6+XvwNvdYo8w/1qXXhChLh3Rgg94mK0Z8OPrhTS+8ydskPcajHM6c7OilbkR57uStv98LhjumL/zbJngPGJjwSg7GYNxkWwFRhtfvQ3qoG1GOT966UWpt6BR0btPs9IUbm6z8FQx96i69EY83eZvk7aRWqgoKetMLQq8VKEm/LiHK4Q6Gt8mKER+OfnjTC2/yNkmP8SiHMydrHXrFo6xACRK69GYUvcnbJCNeP+09KJE3vfDuUZ4paYukj0i6fpBicDwk+/EQiLJ+pqYpBqZ+yDbPiINxeQh4Jsolkq6TtEfSYxDlxIq2Wpy77rZVrdbcjg41aUGU3dPGG+ng7ZRn+Aa15E0vvMnbJD32FHr9gKQfQ5QQZVYzioHJilj28mCcHbOsNcA4K2LZy3v2KNtvewhRTk1NbWy1undrSmvXrs2O0BBr7N23X+u27Ozo8Ygli7X50pULv+3e87Smtj7QUWb50Udq/fkLDqUeeuxJfXz79zvKnHL8Ul295qSF3+55+EeaufvRjjJnnrxME2efsPDbjp27tP2+xzvKrDn9OK1eeewQUaErEAABEKgegbGxsUMcSDzKCsaFXa/DBZ0v8fB4gzEY90LAm1400qPsNTBs5gmzRjk+OWOXP7/dMD9scevwZ/fPPSPp3bPTEzeENxHFevA2We1tvcnsTV4wLjan0tb2pheeifIsSV/tGphl0XrlIeMFUQYlyiu7AL8KokxrMrKVa4qByfbWwy0NxuHxbgrGnkKvqUYVooQouxXF22TF20k11QsX8qYX3uRtkh67Jko28wxvM49tCLKNQfHHNgTZxiAeEAABEGgKAt4386QaBzxKPEo8ylRTpdRCeDulwtmzMTCuDmPXHmUv2CBKiBKiDG9QwBiM0yDgjdw9b+ZJMx4LZSBKiBIjnmnKlFLYm0Fs0vpZKQMYqBFvegFRBlKEPM16PEcZHQ8ZuOt1fP3MDs3p1+KYzM21rr1j8wV35sGprDreJitGvKyRH9yON73wJm+T9Nh16JXNPPXazGNZgixbUPyxLEGWLYgHBEAABDwgwGaemoxSwz3KVV0e5Wo8yuyK58178CZvk7yd7No1vBre9ILQ6/B0I7EniDIRolILeJusGPFSh79vY970wpu8TdJj16HXXjOAzTzVbeaJ1ijxKEuw896Mojd5m2TES1C3YE140ws8ymCqkL1hPMrsmBWp4W2yYsSLjHb6ut70wpu8TdJj1x4lm3nYzJPeLFISBEAABJIRKGMzz6sk3S/JCPa9kuziwz+X9Fhy98MpQei13qHX8ckZu2TzqLg2tOaeW7Nt80UPhtIQvsRDIXuwXTAG414IeNOLskKv35L0LySdL+k2SQ9L+j+SOtalwqtM/x4gShdEeWoXUa6AKDt1uikGpkpbkNQ3GCchVPzvTcE4a+h1n6Qlkm6UdIykiyTtkvSrxSEtpwWIEqLs1iRvk7VJazvlzOowrXjTC2/yNkmPsxLlzyS9VtJdkv69pL+UZBf4Hh5GlbO3ClFClBBl9nlTtAZGvCiCyfXBOBmjoiXKCr3eLOliSU9LeoWkRyXtlLSyqIBl1YcoIUqIsqzZlL4djHh6rPKWBOO8yKWvVxZRWth1dbQ2aeuT9rxe0jfTixK2JETpnyjPW3fL73VrybNHPLdj+6aLn8qjPRiYPKhlqwPG2fDKUxqM86CWrU5ZRFmrXa8cD/F3PGRq6wPavccCEgef9eev0PKjj1z4Yd2Wndq7b39Hmc2XrtQRSxZn03pKgwAIgEBGBMo4HsKu14yg9yo+ygkHouMhA3e9jk/OWGb1jiMk+w9/dikeZQnKF6gJvJ1AwMaaBePqMM66mYddryWMFUQpiDJBj7wZRW/yGvzeZPYmb5MwzkqU7HqFKOMIXDU7PXFD+4c0uV7xKNMpkDej6E3eJhnxdBpVTSlvelHWGiW7XkvQNzxKPMokNWqKgUl6zyr/Dsbh0W8Kxlk9Sna9lqBbEGUxojyA39OtU7qHYvZD/9rW0A95vE1WvJ0SJlmKJrzphTd5m6THWYnS3v35kl4taU7Sd6IzlSnUcjhFOB7i/3hI0maeNB8acW3DwISfe2AMxk34SC0r9Pqbkv5O0j+PQLFk6G+W9O3wapKuB4gSouzWFIx4urlTpBQYF0EvXV0wTodTkVJlEaWFtp6QdKukRZLeHuV5PauIcHnrco5yNM9R2hlLO2sZf+yMpZ215AEBEACBIgiUcY7y55J+XdL/iwR5QXR7CEnRM4xMmtDheetuOXWutciupIo/D85OT6xo/3DuuttWtVpzOzpKtPAo8SgzKGNJRfF2SgJyQDNgXB3GWdcoLW3dJyR9KrqT8ipJ75B0cvhXSNcDoVeIEqJMN1fKLIURLxPN3m2BcXUYZyXKyyOSbEtsG3ouk3RT+FdI1wNECVGapsx75Iuvt/9+6fJfOesfd//T1+y/Z6cv+JfpNKnaUt6Mojd5bXS9yexN3iZhnJUo7d1fF13U/Jyk7ZLuq9akdPYOUUKUB4lycOi6Tnrr3QvGiIfXJjCuDuO0RGk3hgx6OtfJwr9P3x4gSogSohz+BMSIh8ccjKvDOC1RWoh10JO2neBvClFClBBl8Gl2SAcY8fCYg3F1GKcluKTjH7b+U4t7KSFKiBKiDG9QvIeKm7R+NvzRTt+jN3Iv6xzlIITM60xLvOmRzlgSooQo0xLl/PGa5+yIU8czO33hbRnVrvTiTTEwpQNTYoNgXCKYfZpqCsZlEhtEmVLvOEdZQq7XZw6zOyvjz1Oz0xNL2z+UdQ415ZCWXqwpBqZ0YEpsEIxLBBOiTA3m0ImSzDxk5mlrZ3dmnt17ntbU1s58DcuPPlLrz1/I16CHHkvGL7X2UxAEQKARCJSRmWcQEEMnyl7CEHol9Jot9Do4s1FVMx9vJzzyYAzG3QiwRhleJ1L3QOiV0GuSsmDEkxAq/ncwLo5hUgtNwZg1yqSRDvB3iNIPUZ677la7MeeQ547NF34jgGosNNkUAxMSo6Jtg3FRBJPrNwXjMonyeEl27ValD6FXQq9lhl7HJ2eulPTJLqW+YXZ6wvIcB3uaYmCCAVRCw2BcAogJTTQF46xE+QtJlrqu+7FbRewL+72S/nd4+Pv3AFFClBDl8GegN4NoCHmT2Zu8TcI4K1F+WtI5UY5Xq7tGkt1R+RNJb5L0g+i34c/UqEeIEqKEKIc//TDi4TEH4+owzkqUfy/pLZJ2RyIfK+l2SZa5Z5l9pEk6Kvzr4FEaAtxH2aEHQc5REnpNN5sx4ulwKlIKjIugl65uWbte90j6t5Juibq9SNKfSbLsJn8q6Q8kHZdOpDCl8CjxKIftUY5P3rpRaq3q1Oi5O2enL9yYV8u9GUVv8jYpLJhXx4ZRz5telEWUfyPp3V0A/0dJtuHBvMyPSPpQhgEwj/a66E5Lq39FtNYZb+KFkm6OvNYnJL1P0hf69QFRQpQVEeWGLqLcBFFmsAQVFG2KEa8AutRdNgXjrKFXK/+7kl4bIWV3Uf6XaIPPyyT9MDWC8wXfFm0Asn+fJsku2j2xqw37/UWSviTp1ZI+a3fxQpSEXrt0oLLQa+RRQpRjY1ntSUZzUW7xphjxclEpt7WmYJxXsZdIWhSDdG9OeD8syfKMbYnq3x9dCm2eY/dj/b1B0rWSzoYoIUqIMuesC1DNm0E0CLzJ7E3eJmGclSgvl7Q52rgTn25Z22nX/YSkbZJmox/ukvQuSQ/1mMtGxpYI+62S7rG/k+s1OVfpPQ//SDN3P9oB55knL9PE2Scs/LZj5y5tv+/xjjJrTj9Oq1faXq35x9qwtuKPtWFttZ+Pb//+gfyp8efqNSfplOMXcpUfyL9qeVjjj+VftTys7Wfdlp3au29/R5nNl66U5XO1x/5mZeJPqFyvZeH3/V1P9aSnk46tdO9bT5n4EQRGGYEycr3aMZBvS/qKpGdjYE7nBNbWM78n6cao/nejYybtXbXxZs2jfJWkrZIss3VPL5Y1StYoTWnKuj0kza7XNKHX8ckZS1pga/nx56rZ6Ykbes0db96DN3kNc28ye5O3SRhn9QT/V0RW3Vcc5eRJnSfpmujIyRmSjHBtTTL+2IXQh0v6uqQxSXZJ9EskWZKDQx6IEqKEKPNOx/z1MOL5sUtbE4zTIpW/XFm7Xv9IkhGaeXXPxMTZkVO09q5XC+nuknSJpHslLY8I0TYIvUKSJTowb9KOp3ww2vTTs0uIEqKEKHPOxgLVMOIFwEtZFYxTAlWgWFlEaVdp9XqyeqYFXmVwVYgSooQog02vvg1jxMNjDsbVYZyV4Cx9XXxtsi25hUNr8UCUEKVXohyfnGnv/tYZJx5zyb0/+MlNc3OtW+7YfMGdtZhcA4TAiIcfITCuDuO0RLlakoVX7d+9nryh19LfHKKEKN0S5fqZHZpTR4afubnWaoiydDNxoEFvxONN3iZhnJYoLeRqZQm9ljBnuY/Sz32Uw9z1Og5RljC70jfhjXi8yTuKRGlJzy28av/u9RB6TT8/BVFClJG6dBwPgSgzTKISinojHm/yjiJRdqtlWZl5SlD3ziYIvRJ6bXLodXzyNtv13fG05vbftm3zRQ+WPpkyNIgRzwBWzqJgnBO4DNXK2vVadmaeDK+QrihECVE2myhnLOXjqfHZ0Jp7bgVEmc4+xEt5Ix5v8o6yR1l2Zp7s2p1QA6KEKCHK0qdVYoMY8USIChcA48IQJjZQlkdZdmaeRMGzFoAoIUqIMuusKV4eI14cw6QWwDgJoeJ/L4soy87MU/zNulqAKCFKiLL0aZXYIEY8EaLCBcC4MISJDZRFlGUfD0lzcbNdPXFTdMXWTyW9X9Lt/d4YooQoR50ox9fP/GP3/Ni/5NnXbN90ce8rTBLNR3IBjHgyRkVLgHFRBJPrl0WUZWfmSXNx8ysl2T+fk3SKpM9HSdF7vjVECVGOPFFOztilBR33d+0//NmlEGWnyfBGPN7kNbS9yVyUKENl5slycbPhbonRP8rFzfMT/tx1t61qteY6syK1IEqIEqJM9h2aY8TTvGtVZUaNKENl5slycfPLJX0sutPvEUKvEGWXDjw1Oz2xcEO0x/so0yQcGJ9MPh4ynuBRWsKLI/7puSO659Df/eUl/zevQfVmEJvk7eQds2HU86YXRT3KQZl5jpc0kxP0tBc3W8h3UpKd43yi3dfU1NTGVqu1obvvtWvX5hRnONX27tuvdVt2dnR2xJLF2nzpyoXfdu95WlNb7cjcwWf50Udq/fl2Z/X889BjT+rj27/fUeaU45fq6jUnLfx2z8M/0szdj3aUOfPkZZo4+4SF33bs3KXt9z3eUWbN6cdp9cpjF36zNqyt+GNtWFvtx2QxmeKPyWIyLYzZ1gdk7xZ/7J3s3dqPYWMYxR/DxjCyB/zC49cBPv8DAiOEwNjY2CGpXdPmem3DZOV/W5KRo/23fZVeK+m4nDimubj5xZI+JWlcUqeF6NEpa5SEXk0t8Cj7r1GmSaGYdT578xzs/bzJ7E3eJmGclSivl/TO2CSyz367VPmqrBMrKp/m4mYj4u60XUdL+lmvPiFKiBKiHLxGCVHOWw5vxONN3iZhnJUobXv5myV9WdIrItL8YbTBJidXllsNooQoIUqIMo1V8UY83uQdZaK00KctJu2TZJtrnpFkyZgPLkKl0dCAZSBKiBKiLE6UUej6mu6pOjs9cWmv6YsRD2jUoqbBuDqMs3qUd0dHM+xarddGYtsukLHwr5CuB4gSooQoSyPKzt1k0oOz0xMHd5PFpiRGPJ19KlIKjIugl65u0V2v7V7OlPRNSb8RZcixzTz/QdJ30okRvhRECVFClBBlGkvjjXi8yTvKodefS7LzAIm7T9MoaogyECVECVFClGlsizfi8SbvKBPlJkl2XOOLkn4RU8bO7DBptDRQGYgSooQoh0OUlhlqUUsXGt5nnHjMpff+4Cdb7L+3TV9wWaDpXWqz3ojHm7yjTJRlJ0UvVfGtMYgSooQoh0eUSSkUS5/gJTbojXi8yTvKRPlbkn7ZQ1dtc08tHogSooQoIco0xsgb8XiTd5SJ0tLITceU0HKK/UnXb2l0NFgZiBKihCghyjQGxhvxeJN3VInyYkk3S3p7TAlfJmm9pMPTKOYwykCUECVEWR+iHJ+cuVLSq7vn/uz0xB8Owx4M6sMb8XiTd1SJ8tuSXtND8e6KLlWuWu8P9A9RQpQQZe2I8pNdxuGG2emJvGkvS7Mz3ojHm7yjSpT23pau7o0xTbX1SrvN47nStLdgQxAlRAlRQpRpzIg34vEm7ygTZRr9q7QMRAlRQpQQZRoj5I14vMkLUabRworKQJQQJUTpiyjHJ2/dKLX+uNNkzP272ekLN4Y0I96Ix5u8EGVI7S3YNkQJUUKULlHVI/kAABeySURBVImy6wL2uU0QZacxhCgLkkOK6mXlek3RVbVFIEqIEqKEKNNYIW/E401ePMo0WlhRGYgSooQom0eU566bmehlUu7YPDGT19R4Ix5v8kKUeTVzCPUgSogSomweUY5PztgREzuTGX+ump2euCGvWfFGPN7khSjzauYQ6kGUECVECVGmMTXeiMebvBBlGi2sqAxECVFClBBlGvPjjXi8yQtRptHCispAlBAlRDmaRDm+fmaH5vTSuOmZm2v94R2bL7izlznyRjze5IUoKyLBNN1ClBAlRDnSRLmqiyhXQ5RpLGeYMt7IneMhYfQgV6trNnzmqMXPHPZkV+WnZqcnlrZ/O2/dLafOtRY90FXmwdnpiRXt3+zi3KT7AKOk1ANzbUYHvgeeY0uzmSL6oh9oqMYnZ+ydTo2/V2vuuRXbNl/0YPu38UkMfRuLublWh6EHv44Z0bGZJ43+xWs3xYjnMkJDqtQUjFtDwmto3eBR4lHiUfKh0e9DY8OGDYvaf3v72y/Zf/PNN9lVgdq0aVNt8lX3M5beSIfQa3m0Z0R9naTLJO2WdIWkb/Ro/kxJWyR9RNL1g7qHKCFKiBKiLOKRl2feym0JoiwXz16t1TX0+jZJ75Vk/z4tIsETu15gSUSmeyQ9BlESeiV0Teg6shGZQ69pQtfhzXG+HiDKfLhlqVVXovywJFuzMm/Rnvsl2RqXXd3V/XxA0o8hSogSooQoIcos5r+6st7Iva5E+QlJ2yTNRkNpl0C/S9JDaYhyampqY6vV6tqEIq1du7Y6zUjR8959+7Vuy86OkkcsWazNl65c+G33nqc1tbVzL8/yo4/U+vMX9vLoocee1Me3f7+jnVOOX6qr15y08Ns9D/9IM3c/2lHmzJOXaeLsExZ+27Fzl7bf93hHmTWnH6fVK49d+M3asLbij7VhbbUfk8Vkij8mi8nUfuyd7N3ij72TvVv7MWwMo/hj2BhG9oAf+JkehNK/L3/XVoE6H9Pztv6lmOIUcYzA2NjYIXt3qt7M8yFJ35N0Y4TrdyW9KVqv7IYaj1Ji16skPEo8ysg4BAm9Ju26rooDvHlnhpM3mevqUZ4n6RpJb5F0hqTpaK2yly5ClBDlU6YYECVECVFWRdfZ+oUos+HVr3R71+vlknZJukTSvZKWS/qapJdJOkvSV7sasHifrVce8rDrlV2vphScQ9VR8cmx//Bnl27fdDEfGiWd4120d/HBNZAI6Ds2X9hrx345ltKhd4ZHWdrQl98QRAlRQpQcD2lblqoSNpRv2fyFMSHKEFqQo00287CZp602bIZiM1SdNpPlMGdUqQkCddzMUzo0eJR4lHiUeJRVe5QHQv9adFi3gZvdPPGdvEbP23ofHmXekR5CPYgSooQoIcpaEGVCruas5hCizIpY9vJ13fWa/U0SakCUECVECVFClKWb1lwNeiN3iDLXMIepxPEGjjdEmlXJOUD0rx76l9W6eCMdQq9ZRzhQeTbzsJmHzTw6kK2JzEY6kKXKW2aom7/yPzus4/HHPL8js1Yg00mzAxBgM09N1IMv+np80XOfZ8eE4D7UCI6kzDzDnL/xEcKjDG/ACb2Gxzh1D8OcaFzc3DEspAAks5Gri8MhytRmtZSCEGUpMJbTCESJR8ka5cRV7dk0PnnrRqn7coO5TbPTF248WGbmk5Ku7JqBo7LG+9f23ksOW/T8fc8+9wtJt8xOH8SvHKsUphVvXjBEGUYPcrUKUUKUECVEWVboP5cRGlIliHJIQA/qhs08bOZhMw+bedo64HEzTxnX5NXAFDdKBDbz1GQ48SjxKPEo8SjL8ijPWzezppdp27Z5YnvVJg+PsuoR6NM/CQdIOGCqwe0h3B5ielBVUvS66V9V5hqirAr5hH4hSogSoiQzT9tMQJTVGmqIslr8+/YOUUKUECVECVEeumt4TupI0t6Svj47PXFDSFMOUYZEt0DbECVECVFClBBl9uM1Bcxu36oQZQhUS2gTooQoIUqIEqKEKPPQSSPPUXI8hOMh7cnAxc1c3Dyqx0N27Nyl7fc93sELa04/TqtXHrvw28zdj+qeh3/UUWbi7BN05snLFn4z/L6/66mOMu9cdaJOOX5pHs5xW4fjITUZOo6HcDwkUsVRySxjWXXizw3xzDJk5olB06p3RCirCSX0mhWxIZUn9FrviTY+OeMq1ya5cjsmLrlyRzxXblYzDlFmRWxI5SFKiJI1StYoWaPMvkY5vn5mh+a0Km6q+xyvOSFepjX33Ou2bb7owV4mHqIcEvFl7QaihCghSogSogxKlKd2EeUKiDIrU1VcHqKEKCFKiBKirI4o37ju9oXdP3992Wt+9p4b/+EFzzti39z2TRd37hSqmCuyeMCtGspaSCSIEqKEKCFKiLI6oky6+LqQgQ9cuZHHQ3phBlFClBAlRAlRQpR5OLWRRMk5Ss5RticD5yg5R8k5yoPUkPcc5UOPPdnBL1evOanjHOXU1ge0e8/THWXWn79Cy4/Opn/f+sGPD+Gxs099YR5uK70O5yhLhzRfg5yj5BxlpDmco5TEOcqYHan5OcphHe/KZ1mL12qkR0no9bZVrdbcjg4cmGgH4KjbNUfjkzN26P7KLp2FKCFK5u/8pHhqdnpiYRPQ/Pxtnd9t42enL9xUnA77twBRhkQ3Y9t4lHiUeJRc3FzWxc2j6pFnNLupikOUqWAaTiGIEqKEKCFKiLLD3vbwKBdZFq/405EZKoS1hihDoJqzTYgSooQoIUqIEqLMSSHFq3E8hOMhrFFyPKRtSfqkYBuYWSbpHCAfurX60F1z6DrmxHvzMgkeZV7kAtRjotVqorGZou9misGhLzyiYh4R+IXHL6v5hiizIhawPEQJURJ6JfQKUUKUAWlmcNOEXgm9Enol9Erotb6ZeSpwFP6qizVujd+HGv9bIz1KMvOQmaet5GTmyZYZZd2WnR22A/zAb1QzG3W7XmTmqczP7ey4gi8qbpifHwIuHh7xi4eHlVmGhBc6Km719h/+7NL27SF1s3+N9yh78R6hV0KvhF4JvRJ6JfRqOjA+OWPZsAY6ChBlTTzIbjHq9kU1qpk92EwRfjNFGkOF/sXGgRSUB8AYpkcOUUKUqb6oMFQYqt65Njke0taMOocOmb/F5i9ECVFClJ06wBola5SWFo2EA/PzgqT8XRzRyF2vrFFye0hMBzLniiT0SujVEGjNPbdi2+aLHmyjQWaeg3oxCpmN8CjxKPEo8SgPIEDocF4R2PXaMSHwKPEoa8qShL4IfR1UTQwV91GSQnF+PlQWEaqTR9mSdJ2kyyTtlnSFpG90UVm/Mi+UdLOksyQ9Iel9kr7QjwY5HsLxENONYe6aYzNFbDaya/MAGOgf5yjzuGpvk2QZ3e3fp0m6XtKJXQ31K2PlXyTpS5JeLemzkl4KUUqssbHGxhrbxA0La4vrZ3ZoTqviWjEKa2x8qBX7UKuTR/lhSRZy2xIJdb90QKHNQ2w/SWUWSXqDpGslnQ1RQpRdOlBZ6AZDVcxQgR/4EXqd14FPSNomaTZSibskvUvSQzEVSSqzV9KTkt4q6R6rR65Xcr229YdcpeQqHdVcpTt27tL2+x7v+G5cc/pxWr3y2IXfZu5+VPc8/KOOMhNnn6AzT1628Nuo4tftdFWZ6/VDkr4n6cZIqO9KelO0XtmWM6mMeZSvkrRV0gpJRpyHPKxRskZpSsEakZ81IjxKPEo8ynkdOE/SNZLeIukMSdPRWmWc6PqVeb2kwyV9XdKYpK9Jeomkn0OUnKOM6QCh1wgMzgEe1ArWKOexGJ+csXynlvc0/rDruguQqhMOtHe0Xi5pl6RLJN0raXlEfC+zc7/RztjuMq+Q9OnIm9wj6YPRZqBePCk8SjxKPEqSoreNA0QJUZIUndDrqlZrjnNY83qAR4lHya7XBZvI7SGRJ83tIYReCb0SeiUFW1sHxjkeEpsOECVE2TPoOv8joVdCr4ReCb0SeoUoIUqI8gACJBzoUARCr4ReCb0Ser1zdmpi9UKUgYube7MlHiUeJR4lHiUeJR4lHiUeJR7lM4dZ8on4g0eJR4lHiUeJRzmAHxf+hEeJR4lHiUeJR4lHiUeJR4lHiUe5MAu4eHgeCna9xg0jRAlRQpQQJUQJUUpkluE+z8LnyON0UnVmnjRR01LKEHol9EroldAroVc8So8eZZGLm5dKuim6Yuunkt4v6fZ+rApRQpQQJUQJUUKUHomyyMXNr5Rk/3xO0imSPh8lRe/JlRAlRAlRQpQQJUTpkSiTLmW2d0pTxq7Z+igXN8+bARIOdHwrcTwkgoPbQw7qBUnRo41O3B4SNxY3zE5PXNXL06p6jTLpUmaTOanMyyV9LLoq5hFCrxBllw5AlBAl5ygXJgUepUePMulSZnunQWXOkTQpya7geqKtC1NTUxtbrdaGbtJcu3ZtKRuDQjWyd99+rduys6P5I5Ys1uZLVy78tnvP05ra+kBHmeVHH6n159ud1fPPQ489KbuVPP6ccvxSXb3mpIWf7FZzu908/tit5na7efvhhvSD6IDfPBbo3/6OOWNz0+aoPczfpzuwMZtktqn9mG0zjOJPnfHrtvNjY2O2p6bjOeSH7gIl/X+Ri5tfLOlTtgtaUucI9RCONUrWKE0tzlt3y6lzrUWdXxrSg7PTEwtfGoSuOyYQHnkEB6Hrg3oxCqHr+CyoOvRa5OLma6PLmuPvc7Skn/UicYgSooQo2czTtg2jYOjHJ2/dKHVH1gi9mg6MkxS9t68LUUKUECVECVFClBDlgHgwRAlRQpQQJUQJUUKUEOUBBFhjY43NECDX67wekOs1Ph8gSogSooQoyfW6MAsgSoiSNcpoOrSyR9TqtJlnALWV+ydCr9kVhc0AMR3MMdHAD/wksWs4UgNvu4YhynI5uLTW1mz4zFGL8YjwiLj9Yj5kyq7Ng7aFD7UDWAzzeBdEWRq1ldsQRKlT44gSOiR0SOgwf+iQD41iEQ2Islx+K601iBKijJSJ+xTxKAvfpwhRQpSZyYk1StYohx26wVAVM1TgB35VrvHiUWam2eFUwKPEo8SjPHh7A0QJUUKUw+GehV7wKPEo8ShJONA2CKSwi9a/uWYrzkS1vWZraHQJUUKUECVECVGScMB0gFyvfagXooQoIUqIEqKEKCHKAf4pRAlRQpQQJUQJUUKUEOUBBMj12qEIZEaJ4PCWGYXNPGzmYTPP0FYn5zvCo8SjxKPEo8SjxKPEo8SjxKMkBeDCLCCzUbSzc/3MDs1pVdw8sOuVXa+SPtlFGex6xaPEo8SjxKPEo8SjxKPEo8SjxKPEoySp/AEdYI232BpvnE4eeeSRubGxsVY3xRzyw5CXFEvvDo8SjxKPEo8SjxKPEo8SjxKPEo8SjxKPEo+yNVc4qTweZen+ajkNkuuVXK+RJnF7CKHDwoae0Cuh18zMROiV0CuhV0KvhF4JvRJ6JfRK6JXQK6FXQq+EXhsWerVNQ9dJukzSbklXSPpGF98NKnOmpC2SPiLp+kFuJh4lHiUeJR4lHiUepUeP8m2S3ivJ/n1aRHYndhFevzJLIpLdI+kxiHJiRRs3Uth1aBAp7CI4SGF3UC9IODCPxTjXbMWNRW0TDnxY0gORV2gC3y8dyKDxREz6pDIfkPRjiBKiZDMUm6HYDMXF12U5CnXa9foJSdskzUZC3SXpXZIeigmZVOYQopyamtrYarU2xF/0ec97nn75y19m3gREBRAAARAAgdFGYNmyZXrHO95RWcKBD0n6nqQbo2H4rqQ3ReuV7ZFJKtMYjzKuih7WVLunjjeZvclreHuT2Zu8YDwcQvSmF/3kHVZmnvMkXSPpLZLOMDsQrVXGRyupDEQ5HN1O7KUpyp/4ohUWAOPw4IMxGKd1AoZFlO0drZdL2iXpEkn3Slou6WuSXiapX5mzJH2164WWReuVh4w0yo/yp1X+8Ejl7wE9zo9d2ppgnBap/OWagvGwiDI/0hlrNmVgMr72UIuDcXi4wRiM+eALrwNpMW4cUdoGn/Xr128cPsT5evQmr72lN5m9yQvG+eZS1lre9MKbvE3S48YRZdbJQnkQAAEQAAEQGIQARIl+gAAIgAAIgMAABCBK1AMEQAAEQAAERpwo7TjKt2IY/HdJb6yBVqTJf1sDMRdEqCuOvTDqlRu4znj3krfOeC+VdJOkN0j6qaT3S7o9tnN9UE7nKnS6n7x1xviFkm6WZLv+LYPZ+yR9ocYY95O3Thg/P0pyY8luLPlNapswCh7l70h6s6R3VzFDB/SZJv9tnUSuK47dGPXLDVxXvPvJW2e8XynJ/vmcpFMkfV7SS6Jczkk5navQ6X7y1hljy4n9IklfkvRqSZ+V9NIaY9xP3jph/OeSXh/lDjeiTG0TRoEoL5D0Z5KOkfRPEWF+sYrZ2tVnUm7bGojYIUJdceyHU3eCirrj3S2vF7xfJemjks6WVHeMTVfi8nrAeFHkuV/rBONueeuC8UmS/kjSz6Oz+0aUqfV1FIhypST7ZybKBvSfJB0v6dmKmSgpt23F4h3SfV1xTEuUdce7myg94P1ySR+TdKWkRyTVHeNueT1gvFfSk5LeKukeBxh3y1sXjM0jN6L8kxhRptbXphLlX0l6j6THI1KMG9NvS/rdrptLqiClpNy2VciUpc+64JiWKOuOd1KKxrrhfY6kSUmWbat9C1CdMe4lb7fu1A1jk888NPOCt0qyK/Y+mCJvdpZ5XHbZbnmNOKu2v+dL+lVJn47Sp1o2OPMoU+trU4kyPjAWk36BJNvEY5smbBOCxfqfK1tDMraXlNs2Y3PBi9cVx7REWXe8u4myzni/WNKn7JpDSU/HBqCuGPeTt84Ym2yHS/q6pLHIC7J14N9KkTc7uDHo0UE/eY3cq7a/X5ZkH0rxxy7lODItlqNAlKZctnvsdEmPSvoDSXbNV9VPv9y2VcvVr/+64tgtb7/cwD+JFvG78w1XjXc/ee0LuI56a3jZepl5NvHn6ChEeF3kZcZzOleNcT95Tea6YvyKyAMyb9IurTe8rx+QE7tqjPvJWze7YRdytD3K1DZ4FIiyagWifxAAARAAAccIQJSOBw/RQQAEQAAEwiMAUYbHmB5AAARAAAQcIwBROh48RAcBEAABEAiPAEQZHmN6AAEQAAEQcIwAROl48BAdBEAABEAgPAIQZXiM6QEEQAAEQMAxAhCl48FDdBAAARAAgfAIQJThMaYHEPCIQDsRwhpJOzy+ADKDQFkIQJRlIUk7INAsBCDKZo0nb1MAAYiyAHhUBYGKELBLfP84ylVpl/leLek3o3zGfyHp9yT9iqQ/lXRjJOPbJa2X9BuSHpBk90b+ffQ3u6vVLl/+NUn/TdIVUQLur0raIOkdUXtWpt1eRa9OtyAwfAQgyuFjTo8gUASB344I8Q2SfijpXklTkv6HJCO22yT9fpQndLWkX48uV/5mlAD6b6P8pq+LLgewS4G/IenfRDcqfCdKeG53tlp7t0T5kf+zpFMl2U32PCAwUghAlCM13LxsAxAwj9E8w/jzOUl2tZwR21skGcldJMnu4DNCNMLcJMmSgP9M0oSkWyMv1AjXLjZfKumpWKPt0KvdAv95SXa7iZWzGy32NQBHXgEEUiMAUaaGioIgUAsE2jdh9CO2NlFeHHmOZ0iyDTl2+4SFVu0mijaJ2tVIRqIbo6uQ7ILg9tO9Rml3T5rnalcTdd8xWAtgEAIEQiEAUYZClnZBIAwC/0rSV6Lb2s1ztPsKzcu8P/IozVN8Z3TvqnmLFnq1K5BsPdJud/9k5E1ayPVESa+VdLekd0m6Q9K3JNm6p10/ZR5qe9crRBlmPGnVAQIQpYNBQkQQ6ELgmogo/1kUFjViNM/RiO1votDqs9EGHVtjtMfu4TSyO07SP0h6j6T7or/Z5iDb3GOhWbur9UpJJ0OU6B0IzCMAUaIJINAMBNqhUgul3tmMV+ItQKAeCECU9RgHpACBoghw7rEogtQHgT4IQJSoBgg0AwGIshnjyFvUEAGIsoaDgkggAAIgAAL1QQCirM9YIAkIgAAIgEANEYAoazgoiAQCIAACIFAfBCDK+owFkoAACIAACNQQAYiyhoOCSCAAAiAAAvVB4P8D+77+ovte0PAAAAAASUVORK5CYII=",
      "text/plain": [
       "<VegaLite 2 object>\n",
       "\n",
       "If you see this message, it means the renderer has not been properly enabled\n",
       "for the frontend that you are using. For more information, see\n",
       "https://altair-viz.github.io/user_guide/troubleshooting.html\n"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import altair as alt\n",
    "\n",
    "df_loss = pd.DataFrame(enumerate(training_losses), columns=[\"epoch\", \"training_loss\"])\n",
    "alt.Chart(df_loss).mark_bar().encode(alt.X(\"epoch\"), alt.Y(\"training_loss\", scale=alt.Scale(type=\"log\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And because we're paranoid types, let's check a prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 4.0063, -7.1900, -6.8099, -5.7869, -6.4321, -3.4320],\n",
       "        [ 4.9544, -6.3152, -7.6040, -6.6827, -7.8661, -5.0460]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    logits = model.forward(**next(iter(dataloader)))\n",
    "logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The positive sample gets a positive score and the negatives get negative scores. Super."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should be able get the paragraph vectors for the documents and do things like check these for similarity to one another."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_id</th>\n",
       "      <th>similarity</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>It was a warm night at Castle Caladan, and the ancient pile of stone that had served the Atreide...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0.177416</td>\n",
       "      <td>In the week before their departure to Arrakis, when all the final scurrying about had reached a ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>0.081760</td>\n",
       "      <td>By the half-light of a suspensor lamp, dimmed and hanging near the floor, the awakened boy could...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>-0.044768</td>\n",
       "      <td>The old woman was let in by the side door down the vaulted passage by Paul's room and she was al...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   doc_id  similarity  \\\n",
       "1       1    1.000000   \n",
       "0       0    0.177416   \n",
       "3       3    0.081760   \n",
       "2       2   -0.044768   \n",
       "\n",
       "                                                                                                  text  \n",
       "1  It was a warm night at Castle Caladan, and the ancient pile of stone that had served the Atreide...  \n",
       "0  In the week before their departure to Arrakis, when all the final scurrying about had reached a ...  \n",
       "3  By the half-light of a suspensor lamp, dimmed and hanging near the floor, the awakened boy could...  \n",
       "2  The old woman was let in by the side door down the vaulted passage by Paul's room and she was al...  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import normalize\n",
    "\n",
    "def most_similar(paragraph_matrix, docs_df, index, n=None):\n",
    "    pm = normalize(paragraph_matrix, norm=\"l2\")  # in a smarter implementation we would cache this somewhere\n",
    "    sims = np.dot(pm, pm[index,:])\n",
    "    df = pd.DataFrame(enumerate(sims), columns=[\"doc_id\", \"similarity\"])\n",
    "    n = n if n is not None else len(sims)\n",
    "    return df.merge(docs_df[[\"text\"]].reset_index(drop=True), left_index=True, right_index=True).sort_values(by=\"similarity\", ascending=False)[:n]\n",
    "\n",
    "most_similar(model.paragraph_matrix.data, example_df, 1, n=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's not particularly illuminating for our tiny set of dummy data though. We can also use PCA to reduce our n-dimensional paragraph vectors to 2 dimensions and see if they are clustered nicely."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2-component PCA, explains 48.18% of variance\n"
     ]
    },
    {
     "data": {
      "application/vnd.vegalite.v2+json": {
       "$schema": "https://vega.github.io/schema/vega-lite/v2.6.0.json",
       "config": {
        "view": {
         "height": 300,
         "width": 400
        }
       },
       "data": {
        "name": "data-0b1e0474027b1da095c3ed8d1f5f386e"
       },
       "datasets": {
        "data-0b1e0474027b1da095c3ed8d1f5f386e": [
         {
          "group": "0",
          "x": -4.767539597377173,
          "y": -4.756888669464686
         },
         {
          "group": "1",
          "x": -3.202457816084362,
          "y": 6.363203538611249
         },
         {
          "group": "2",
          "x": 6.650774519184007,
          "y": -0.034075790016020964
         },
         {
          "group": "3",
          "x": 1.3192228942775253,
          "y": -1.572239079130542
         }
        ]
       },
       "encoding": {
        "color": {
         "field": "group",
         "type": "nominal"
        },
        "x": {
         "field": "x",
         "type": "quantitative"
        },
        "y": {
         "field": "y",
         "type": "quantitative"
        }
       },
       "mark": "point"
      },
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfwAAAFZCAYAAAB9g51OAAAgAElEQVR4Xu2dC5RdVZnn//tWeBiwfXRjsHmksRCVbnSle2xsF4M4tEIgCYgWCciAoEtcOgpCgEqQR6NQxSug2LToNAIuQyoKYsLDHh/YIz7QaekR2wEGpEEeAW3FHl4tqXtm7eRWrNzcx77n3HPvP7d+tRaLBbX3Od/5/7/v++297711g/hBARRAARRAARQYeAXCwD8hD4gCKIACKIACKCCATxKgAAqgAAqgwAxQAODPAJN5RBRAARRAARQA+OQACqAACqAACswABQYO+GvXrs323nvvGWAdj4gCKIACKNBtBYaHhweOi1MaDdyDjY+PZ6Ojo319rgceeCBzSBqHOIjh9+0ILTZq4aCDSxwOWjjEgB/dXrY0vl5fwVjGIwJ8AFOfVzQ0cqJRr3HIC2IgN8vgYLNrAvwS1HYoYlbMXo0EP/CDRUfrZuvQN7sYQ2TrpyQdI+k+ST+TtFjSbElvlfQtSedJOlXSXpIOlLRM0m6SfirpZEl3Sjpc0lckzZP0z5Iul/Q+STtK+i+SvinpAklHStpB0nJJ1wD8EsDe7JJdTJpCUTvEQQxeoMMP/OAErHlb7WJ9HCLplhq4V0v6nqSdaqDeT9J3JH1B0gck7SPp+5JOl/TZ2v/fV9KrJL29BfCnrjMh6f2SPi/pYEmvlPTvjZ6SHX4hpDae3MWkKRSdQxzEAGAATE8Ak7tXONToAJ6AnSPpXEl/JOnfJF0p6dg64L9T0o2Szpb0N5JeJukpSUskXS/pzZLmJAD/MElrJB0t6YuS4mLhhwA/d0l0NpECAnIc33pDbgAB01mTmjaaflVKv0oB/nxJX5N0Vu14/+WSfjMN3G+StLOkmyT9J0n/JOkqSe+uWzhMAT++fBBPDd4o6X8B/Nwl0dlECqiUAurMBBpaQ70cctMhBoDvVaMD6MfC2q77I5K+3ORIfwr4EdDx9frTJH2utrt/g6RX14774+/i6/sR5t+uHdnH1/CnjvTjacCJkq6rvRcgHuk/A/BzI6OziTQ0r2aCH/jBiQsnLildvIu9YqgG73hsH9+E96ik+Lr+dFBPAT+GdrykUUm71N6cd1JtRx9/F0F+hKS7aguDD9be/DcF/CtqLwOsl7RU0spmz8pr+ClZ0OGYLiZNh3fefLhDHMQAbOuT2CEnBnBHmbtX4EdPavTvJL2jdkSf26u6iVPAj2/U+4eUiwL8FJU6HEMB9aSAkl3BD/xgh88OP6VhdLFXxOP4/1n7aF48po/vwv+X2sfsUkJJGTMF/OknBS3nAfwUWTsc08Wk6fDO7PBdGzs7Sq9FB37gRw96RXzdPR7Nx8/H31F7nf3hQk1988kAf6b8pb1nx+YcFar6k+h/NejJUKl8ffbo45slk8PCgxi8Git+4Ec9cBxyggVYF5cBLS7FDr8EncsuoGc+sfPaLcIOejgMhQunQ7/sOFKkIwYAA2C8j9MdahTgp3TT4mMAfnENt7hCmQUUd/bZZDg6DGUrq1U9Wc2y54ZC5UBl+svJrPr3f3DWk/Ezmxt+yowjVTZiAPgAH+Cn9At6RYpKxcYA/GL6NZxdZuI+e8HOS7Oq3hKUnT/7Y0/8IAawaRFQ0T/OXr7uEoC/uS1l+tFJ+jjEQQwswFiAeS/AOukpnY4F+J0qljC+zKb63PlzllWz8OahoXDF9sse/x8xnOfO3/nYaqaRUAm3z17++AqAD/CbpWmZuZlQGjYnT5yAeS18BtmP+aetGp5V0V7Vil4Tn7NS1b3rq7rvtouXPJBaM90aB/C7peS065TZVJ+5YM57VQ2HK4S7K8ruCZWwbrKqA5Rl+yjoyh3OXHcbwAf4AL99YZdZp+3vvnEEMXgtPLrtx8JlqxZkWThByn4nhSc2Pm02RwrbhpBdvXZsyc2pudKNcQC/GyrWXaPbSTP98s+Ov3L3bDI7Q5l2n/7/Q9Ads89cd+H0/1dmHKmyEcNgN7TUPHDLS2DrlZeD6Mei0evnVRXOCwp3rB1fvFlvXjg6cUambL+KsrPXjB8V/4Je/U9k82WS3iNpnaQTan+eN0/JbZoD8AvJ13hy2ZCL0F//wuSfzxqq7DmZZf8RpJ/teOYT8XuRN/spO44U6YjBq7HiB3449olBA/4hZ94wt7J+/SUh6Ma144vj37rf4mfh6MRRWaYjqrNmLb31/Hc+VDcg/lW+k2t/nW+epM/U/rZ+StttOgbgF5KvP8BPDZnmvlEpBx1c4nDQwiEG/PBa+AyaHwtPX3l4VqksuXl8Sfyq26Y/C0ZXrQrV6qq1Fx296dNVtcEX1/4G/7W1//6JpIMkPZ7a/xuNA/hF1Gsyl4bm1UzwAz8alapDXhDDYObmwmUTS7Msm3vz+JIPtwH+FSGEh9aOLd706ara+Pg1uPHvrUy9xv8tSfFLc+4pgqytAfjHSTq/9u1A8TuG4zcDNf2ZKX9pL8V0mgk7/Po8IScGEzAp/aDZGIecGLQd/qHLVp0csrDHzeOL45/WbbHDn/hkFrIHbxlbcnndoIsk/UzSNbX/f7ekt9Vez89ttzvwd5b0dUnxu4Xj9/u+qvb1gAA/wXKHQiYGAOO46Bg0wCS0g6ZDHGp00PxYeMaq+VnQcUlH+pmuXXvhkk2frqoZFZl3uqTDJL1R0rik+Fp+oR934B8j6VBJb5K0vaR4PPJldvhpnjsUMjEAfIDfvF6pj8Gsj/jZ+6GhcHGQ7qx/h/7UE298p772nZzMTmvwmfypd+kfL+kxScdK+lFa528+yh34SyUtkPQuSXtIim9s2C1+X0x8pLGxsXNDCPGYf7OfkZGRorowHwVQAAVQYAYqMDw83BUubvwMvk4MCivr36m/4R36yo4OQVf18rP4XXmwEnPiREm7Sjqrdo8fSzpY0pPN7slr+IO5Yi6SYw67qEE7ssSPIgpQo43Uc6jTbsew4bP4WWW5QjYp6Ze1595JWRiqhOoFTT6D353kanAVd+DvLekGSX8taY6kr9R2+ht2+I1+AD7NpD4vul3EeavRIQ5ioD6oj+YVXEZ9xM/kD73wH/M0NLTnhjtPTt4/uc12dzX47H3e1pI8zx348UHisX5888Jzkj407WMKAL+NzWUkb3Jm1QYSA4ABML0FzNZYo5yAdepavvFbA/A7ejJ2+AAGwACYlKbBYnSjSg46uMThokVK/uYZA/DzqLYV7KwpIK+FD37gR6O24QAYhxiojxJA1OCSAL8EnSkgr+aOH/gBbL1PfQB+CSAC+L0RFcAAGAADYFK6jUOvcIhhkIH/1MdfMbxNpbJXloXXxOcMIbv3hWr1vpee9eQDKTnSzTHs8LupZu1aFBDAB/gAP6W1OPQKhxgGFfjPfHzOAlV0QhbC7yoKT8TnrCqbE7JsW1V19Q5nPTH1t/JT0qXwGIBfWMItL0ABAXyAD/BTWotDr3CIYRCB/8zYH8/TZPW8EHTH7DPXXTg9H549f+czskz7aahy9g7LHrurSa78laT4bXkral+Pm5JSLccA/MISAvxWEjo0E4cYBrGh5S0d/GBBPOgL4uc+8cq51ZBdEirZjbOXPXF9o+d9dmzOUVk1HFHJwtIXfezxh+rGbCvpMkm/kfQIwG/SbfhYHs2kPjUADDkx6IDJu/hyWYi6xNGtXvHvH3/F4bOGKktmL1+3pJU3z16w86r1k9VVf3DWk/HPxjf6+ZikXwF8gN+2xruVvG1v1GIAMQBbFmDNC4T6GMz6ePaCneMfjJs7e/m6+IVvTX+evWDn+HXvD81evu4SgJ+DNOzwB7OAcqTCpikOTXXQdjD4UUQBanTQT1ye/sQrT65UtMfs5Y+f1Br4r/xktaoHd/zY45cD/Bw1BfBpJuwo2VGmtA6HhSAxDGa/eub8nednQcft2OZI/+kLdl4VMl27w5nrbgP4KVVbNwbgD2YB5UgFdvgNRAMw1AcL4vIXxBs/ex8uDgp31r9Df+ruG96pr2zfF6rZaQ0+k7+fpO/URbpT7fX83O2Qd+nnlq78pCkaGs19o4IOOrjE4aCFQwz44bXwGUQ/NnwGP4QTw1C2sv6d+hveoT8ZjlaWXdXLz+ID/KJUNd1FDWIB5bUKwHg1d/zAj0a17JAX3Y5hw2fxq9XlIWhS0i9rz71TlmlIlcoFLT6Dn7fdtZwH8EuQtdtJkzdEhziIgebOEbL3aaBDjQ7yBiV+Jv+FbHLerKHKnvE5109W798mDN3V4LP3eVt98jyAnyxV+kAKCMjNlB1MelV45cQgA6ZTT+hXfrnZqYep4wF+qlIdjKOAvAoIP/CDBZj3KQMLsA4AU2AowC8gXrOpAAbAABgAk9JaHHqFQwwAPyVbio8B+MU13OIKFBDAB/gAP6W1OPQKhxgAfkq2FB8D8ItrCPBbaOjQTBxioKF5LQLxAz96tSh/ePHi4SDtFaTXxHtm0r2ZdN/uExMPlICflpcE+CUoDmC8mgl+4Eevmnun7cQhNx1iGNQF2COLFy8IWXZCFsLvQsie2AD8LMwJWbZtFsLVu05M3NxpzhQZD/CLqNdkLgUEYAAMR/oprcWhVzjEMIjAf3TJknlZNnleCOGOXVatvnB6Pjy65MgzsizbL4Shs3dZtequBrnyEknXSTpQ0q8lxS/jWZ2SU63GAPyiCjaYTwEBfIAP8FNai0OvcIhh0ID/2NFHz62uX39JGNKNu1y/+vpGufDoUUcelU3qiMqsWUv/eOXKh+rG7CMp/nODpNdKil+fu0dKTgH8oip1OJ8CAvgAH+CntA2HXuEQw6AB/+Ejjzx8KIQlu0xMLGmVB48uXrxqMstW7b56dQR6s5/XS/q0pP1TcgrgF1Wpw/kUEMAH+AA/pW049AqHGAYN+I8uOXKpsjB3l4mJD7cB/hUK2UO7rFp9SZNxr5N0paT3xa8FSckpgF9UpQ7nU0AAH+AD/JS24dArHGIYNOA/snjxySFke+yyavVJLYG/5MhPZll4cNeJicsbjDtA0qik4yU9npJP7cbwGn47hXL8ngIC+AAf4Ke0Dode4RDDoAH/F4sXzw/Scbu2OdJ/ZPHiVZl07W4TE7fV5ctcSVdLWiDpuZRcShkD8FNU6nAMBQTwAT7AT2kbDr3CIYZBA3787P1Qll2siu6sf4f+VF7Ed+qrqn0nQzitwWfyz5J0Xl0OvUzSUyl51WwMwC+iXpO5FBDAB/gAP6W1OPQKhxgGDfjxeeJn8KXqiaESVta/U3/DO/Sr2dFS5apefhYf4KdUZYdjKCCAD/ABfkrbcOgVDjEMIvDjM234LH61ujyEMCnpl7Wc2CnLsqFQqVzQ5DP4KamTawzAzyVb60kUEMAH+AA/pbU49AqHGAYV+PG54mfy169fP2+ooj3jf09Wdf+sWbPuavDZ+5SUKTQG4BeSr/FkCgjgA3yAn9JaHHqFQwyDDPyUPOjVGIBfgtIUEMAH+AA/pbU49AqHGAB+SrYUHwPwi2u4xRUoIIAP8AF+Smtx6BUOMQD8lGwpPgbgF9cQ4LfQ0KGZOMRAQ/NaBOIHfrguyktA0qZLAvwS1AUwXs0EP/DDtbk75KZDDIO8ADvt1q8MV4eG9som9Zr4nGFI91YmJ++7+JB3FP5TuZ3iC+B3qljCeAoIwAAYjvQTWoUceoVDDIMK/FNuWbNAlewEZeF3CuGJDTmRZXMUsm1VDVevOHTRzSl50q0xAL9bSk67DgUE8AE+wE9pLQ69wiGGQQT+Kbd+dZ4UzlNFd6w4eNGF0/PhlK+tiX9lbz8pO3vFIYfd1SBX5kj6ghTHbPg7+qdI+mpKTrUaA/CLKthgPgUE8AE+wE9pLQ69wiGGQQP+qbfcMrdaWX9JReHGS+cvur5RLpx625qjqsqOqFRnLb300EMfqhszT9Lukr4u6Q2SvijpVSk5BfCLqtThfAoI4AN8gJ/SNhx6hUMMAwf8W9cenlW0ZMXBC5e0yoNTvrZ2Vahq1aWHLLypybiKpAMlxb+tv39KTgH8oip1OJ8CAvgAH+CntA2HXuEQw6AB/5Rb1yxVJcxdcfDCD7cB/hWqZg+tOGTRJU3GPS/pt5IOl/T9lJwC+EVV6nA+BQTwAT7AT2kbDr3CIYYBBP7JCmGPFfMXntQS+Let/aSy7MEVhyy6vMUO//WSvizpzyTFBUDuH17Dzy0dDS1FOodm4hDDoDW0FO+bjcEPFsSDviA+9R/WzM8mddyKQxa1PtK/dc2qMKRrLz1o0W11mrxJ0naSvitpWNIdkvaQ9HSR2ttagD9b0j2SPiip5ccYxsfHs9HR0b4+Fw2NhjboDa1I06E+qI9Br4/42fvJULlYIdxZ/w79qWff8E79LNt3KKue1uAz+XtL+rykuLv/jaTzJH2mSN3FuX0FYwfBny8prnguA/jpqjk0VmKguddnrENOcOLilZeD6MeGz+AHnRiCVta/Uz++Qz/LdLQyXdXLz+JvDcDfS9JHakcZ8ViDHX4i8x0aKzF4NVb8wA8WYL17OTZ+Fj8LWh5CZVLSL2t33inLqkMh0wVNPoOf2OE7H7Y1AD9+/jAC/7Ta6xibgD82NnZuCOGc+sceGRnpXAlmoAAKoAAKzHgFhoeHu8rF+Jl8heq8TNmeG4/Vw/3KKnc1+Ox96dp39cFKiPZdkl5cey1jvB74je7Ha/jsYNjB9G4Hk6fmHU4ZBvEIOY8XLjq4xOGSm3m9bDfPHfi3Szqg7iHeJukbzR4M4AN8gA/w2zU+AOPVJ/AjJWOLj3EH/vQnZIffod8Oq1Vi8Gqs+IEfLIi9F8QdtvmOhm9NwE96MHb4NDQamndDc1h0sKP06hP4kYS3woMAfmEJt7wADc2rmeAHfjQqc4e8IAZyswQENb0kwC9BbYciZsXs1UjwAz9YdLRutg590yGGEpC06ZIAvwR1XZLGIQ5i8AIdfuAHL3l5v+RVApIAfpmiOjRVdpRejR0/8IMdPjv8MrmTcm12+CkqdTgG4Hs1d/zAD2Drv6t1qFOHGDrETUfDAX5HcqUNdkkahziIAdhyhOwNW4ca5QQsjS1FRwH8ogo2mE8BATl2lN6QAzBeNYofJYCowSUBfgk6A3yvZoIf+MECjAVYSqt36RUpseYZA/DzqNZmjkvSOMRBDMCWI31v2DrUKDv8EkDEDr83olJAQI4dpTfkAIxXjeJHb9jEDr8EnQG+VzPBD/xgAcYCLKXVu/SKlFjzjAH4eVTjSD9ZNYcCcoiBHYzXogM/8MN1EZjcXHMMBPg5RGs3BcB4NRP8wA/X5u6Qmw4xsABrR5Xu/B7gd0fHza5CAQEYAMMRckprcegVDjEA/JRsKT4G4BfXcIsrUEAAH+AD/JTW4tArHGIA+CnZUnwMwC+uIcBvoaFDM3GIgYbmtQjED/xwXZSXgKRNlwT4JagLYLyaCX7gh2tzd8hNhxhYgJUAogaXBPgl6EwBARgAw5F+Smtx6BUOMQD8lGwpPgbgF9eQI32O9JOyyKGxEgOL0fpkdcgJgJ/UQgoPAviFJdzyAhQQTZUdPjv8lNbi0CscYgD4KdlSfAzAL64hO3x2+ElZ5NBYiYHFKDt8/8VoUkPJMQjg5xCt3RSHpsqK2aux4wd+cOrTunM69E2HGNrxpcjvAX4R9ZrMdUkahziIwQt0+IEf7PDZ4ZeAvf5ccnx8PBsdHe3rQsahqbKj9Grs+IEf7PDZ4feHir+/a1/BWMbDA3yvxuqw+HGIAeB75SV+4IfrAqwMLk5dE+CXoC6A8Wom+IEfrs3dITcdYmABVgKIGlwS4JegMwUEYACM/+ukDnVKDPSKEhDU9JIAvwS1HYqYFbNXI8EP/GARyGv4JeCmo0sC/I7kShsM8L2aO37gB7DlxCWle7v0ipRY84wB+HlUazPHJWkc4iAGYFtfLg45wYmLV17iRwkganBJgF+CzjQ0r2aCH/jBDp8dfkqrd+kVKbHmGQPw86jGDj9ZNYcCcoiBHYzXogM/8MN1EZjcXHMMBPg5RGs3BcB4NRP8wA/X5u6Qmw4xsABrR5Xu/B7gd0fHza5CAQEYAMMRckprcegVDjEA/JRsKT4G4BfXcIsrUEAAH+AD/JTW4tArHGIA+CnZUnwMwC+uIcBvoaFDM3GIgYbmtQjED/xwXZSXgKRNlwT4JagLYLyaCX7gh2tzd8hNhxhYgJUAogaXBPgl6EwBARgAw5F+Smtx6BUOMQD8lGwpPgbgF9eQI32O9JOyyKGxEgOL0fpkdcgJgJ/UQgoPAviFJdzyAhQQTZUdPjv8lNbi0CscYgD4KdlSfAzAL64hO3x2+ElZ5NBYiYHFKDt8/8VoUkPJMQjg5xCt3RSHpsqK2aux4wd+cOrTunM69E2HGNrxpcjv3YH/EknXSTpQ0q8lLZW0utUDj4+PZ6Ojo319LpekcYiDGLxAhx/4wQ6fHX6RRUOZc/eRFP+5QdJrJd0kaQ+AnyY5zX2jTg46uMThoIVDDPjhtfDBj7SeXnRUX3fCHQb/ekmflrQ/wE9TzqGxEoNXY8UP/GCHzw4/jSD9G/U6SVdKel/csE2FMTY2dm4I4Zz6sEZGRvoXKXdGARRAARTYahUYHh7emjbCHem8NTzYAZJGJR0v6fF2T8dr+Oxg2MF472AcThk4QvbqE/jRjmzd+b078OdKulrSAknPpTwywPcqZIfm7hADDc0rL/EDPxrxxKVXpLAuzxh34J8l6by6B3uZpKeaPSzA9ypkhwJyiAHAeOUlfuAHwM+zZDCbA/C9CtkBtg4xABivvMQP/AD4ZvDOEw7A9ypkB9g6xABgvPISP/AD4OchrNkcgO9VyA6wdYgBwHjlJX7gB8A3g3eecAC+VyE7wNYhBgDjlZf4gR8APw9hzeYAfK9CdoCtQwwAxisv8QM/AL4ZvPOEA/C9CtkBtg4xABivvMQP/AD4eQhrNgfgexWyA2wdYgAwXnmJH/gB8M3gnSccgO9VyA6wdYgBwHjlJX7gB8DPQ1izOQDfq5AdYOsQA4Dxykv8wA+AbwbvPOEAfK9CdoCtQwwAxisv8QM/AH4ewprNAfhehewAW4cYAIxXXuIHfgB8M3jnCQfgexWyA2wdYgAwXnmJH/gB8PMQ1mwOwPcqZAfYOsQAYLzyEj/wA+CbwTtPOADfq5AdYOsQA4Dxykv8wA+An4ewZnMAvlchO8DWIQYA45WX+IEfAN8M3nnCAfhehewAW4cYAIxXXuIHfgD8PIQ1mwPwvQrZAbYOMQAYr7zED/wA+GbwzhMOwPcqZAfYOsQAYLzyEj/wA+C3JmyQlOWBcC/nAHyvQnaArUMMAMYrL/EDPwB+azI/IOkaSddKeriXEO/kXgDfq5AdYOsQA4Dxykv8wA+A35qsqyUdJGlHSd+S9HlJN0p6vhMglz0W4HsVsgNsHWIAMF55iR/4AfDb03gbSftLWiDpGEnxvz8r6TxJT7efXv4IgO9VyA6wdYgBwHjlJX7gB8Bvz+M/lXSEpHdK2kfSt2v//rGkg9tPL38EwPcqZAfYOsQAYLzyEj/wA+C35vG9kvaS9FjtOP9zkh6StJuk/ytp+/Jx3v4OAN+rkB1g6xADgPHKS/zAD4Dfmqe3SIqQXytpctrQIUknSrqyPY7LHwHwvQrZAbYOMQAYr7zED/wA+OXzuPQ7AHyvQnaArUMMAMYrL/EDPwB+6Tgu/wYA36uQHWDrEAOA8cpL/MAPgF8+j0u/A8D3KmQH2DrEAGC88hI/8APgl47j8m8A8L0K2QG2DjEAGK+8xA/8APjl87j0OwB8r0J2gK1DDADGKy/xAz8Afuk4Lv8GAN+rkB1g6xADgPHKS/zAD4BfPo9LvwPA9ypkB9g6xABgvPISP/AD4JeO4/JvAPC9CtkBtg4xABivvMQP/AD45fO49DsAfK9CdoCtQwwAxisv8QM/AH7pOC7/BgDfq5AdYOsQA4Dxykv8wA+AXz6PS78DwPcqZAfYOsQAYLzyEj/wA+CXjuPybwDwvQrZAbYOMQAYr7zED/wA+OXzuPQ7AHyvQnaArUMMAMYrL/EDPwB+6Tgu/wYA36uQHWDrEAOA8cpL/MAPgF8+j0u/A8D3KmQH2DrEAGC88hI/8APgl47j8m8A8L0K2QG2DjEAGK+8xA/8APjl87j0OwB8r0J2gK1DDADGKy/xAz8Afuk4Lv8GAN+rkB1g6xADgPHKS/zAD4BfPo87vUOQdJmk90haJ+kESd9rdRGA71XIDrB1iAHAeOUlfgy2Hw+NjOw3FMK+CuGNku4Oleznu1y/+vp2AHLpFe3izPv7CFTnn3dIOllS/Pc8SZ+R9GqAn2aZQ/ISg1djxQ/8qO8eDjnRzQXYYyMjf1Gt6Nz65wyVsLId9F20SOvwnY9yB/7Fkn4q6drao/1E0kGSHm/2qOzwaWiD3tA6L3NywvX41gEwDjF0C/gPjIy8ZLtK5SJJ22bV6t/9Trp3O2k/VfQBKTy868TqD7WqHxctitR4q7nuwL9K0lpJN9ce4luSPijpnvjfY2Nj54YQzql/wJGRkbL04roogAIogAKmClTuvVdDd/5A1bl/osm3vGVjlM8/r21WT0hDQ3rh3ce0jXx4eNidi22fodkA9weLK7WfSbqm9gB3S3pb7fX8hs/EDp/dHDv85v3AYQfjEEO3dpS5O29tooMWDjF0y49fjIzsEyq6IITw/3ZZtfroeN1fjIzsEirx5eDw6K4Tqz/ADr9o1pY3f6Gk0yUdJim++WK89lp+0zsCfIAP8AF+SktyAB0xdL9fPbL4yL+Vst3jEX6W6Z+DNEch21eqfHPXiYnLAX5KdfRnzNS79I+X9JikYyX9qFUoAL/7BVTEehoafrAAYwGW0kO61SseO+qdr82yobOzLHvxtPvet+vEl05tF0e3Ymh3n3793v1Iv2NdAD6AATAAJqVxODR3YiinX2UjI9s+OjT0n7Nq9aUhyx7b9Utf+v7WkhMpceYdA/DzKtdinkMRx/Ac4iCGchpa3rTFD/xgQey9IM5b2ynzAH6KSh2OcWiqAHv45SQAABTzSURBVN+rseMHfjRqIw69wiEG6qNDyOQcDvBzCtdqGgXk1dzxAz+Arf+u1qFOHWIoAUmbLgnwS1DXJWkc4iAGYMsRsjdsHWqUHX4JIGpwSYBfgs4UEJBjR+kNOQDjVaP4UQKIAH5vRAX4Xs0EP/CDBRgLsJTu79IrUmLNM4Ydfh7V2sxxSRqHOIgB2HKk7w1bhxplh18CiNjh90ZUCgjIsaP0hhyA8apR/OgNm9jhl6AzwPdqJviBHyzAWICltHqXXpESa54xAD+PahzpJ6vmUEAOMbCD8Vp04Ad+uC4Ck5trjoEAP4do7aYAGK9mgh/44drcHXLTIQYWYO2o0p3fA/zu6LjZVSggAANgOEJOaS0OvcIhBoCfki3FxwD84hpucQUKCOADfICf0loceoVDDAA/JVuKjwH4xTUE+C00dGgmDjHQ0LwWgfiBH66L8hKQtOmSAL8EdQGMVzPBD/xwbe4OuekQAwuwEkDU4JIAvwSdKSAAA2A40k9pLQ69wiEGgJ+SLcXHAPziGnKkz5F+UhY5NFZiYDFan6wOOQHwk1pI4UEAv7CEW16AAqKpssNnh5/SWhx6hUMMAD8lW4qPAfjFNWSHzw4/KYscGisxsBhlh++/GE1qKDkGAfwcorWb4tBUWTF7NXb8wA9OfVp3Toe+6RBDO74U+T3AL6Jek7kuSeMQBzF4gQ4/8IMdPjv8ErDXn0uOj49no6OjfV3IODRVdpRejR0/8IMdPjv8/lDx93ftKxjLeHiA79VYHRY/DjEAfK+8xA/8cF2AlcHFqWsC/BLUBTBezQQ/8MO1uTvkpkMMLMBKAFGDSwL8EnSmgAAMgPF/ndShTomBXlECgppeEuCXoLZDEbNi9mok+IEfLAJ5Db8E3HR0SYDfkVxpgwG+V3PHD/wAtpy4pHRvl16REmueMQA/j2pt5rgkjUMcxABs68vFISc4cfHKS/woAUQNLgnwS9CZhubVTPADP9jhs8NPafUuvSIl1jxjAH4e1djhJ6vmUEAOMbCD8Vp04Ad+uC4Ck5trjoEAP4do7aYAGK9mgh/pfpxz++3b//aZ3+xRGdr21Vl4YZ30wv2XHXzkr9vlfCe/x490PzrRNe9Y/PDyI6+PKfMAfopKHY6hgLwKCD/S/PjImjW7b7ONzsgy7T495UPQhZfOX3RHh2XQdDh+pPnRLb3bXQc/vPxo51eR3wP8Iuo1mUsBeRUQfqT5ccqtay+Usr0Vws+VVX+qSthbVe0ZZ4fJcPylCxf+qhvlgh9pfnRD65Rr4IeXHyme5R0D8PMq12IeBeRVQPiR5scpX/vqtaqGl88K2dKL5h92b5x16m1rr8+ybMdu7vLxI82PElpTw0vih5cfZfoO8EtQlwLyKiD8SPPjlNvWrFamF1WGtltyyUEHPRNnffRrXz01VMMBCpXPrZi/YE03ygU/0vzohtYp18APLz9SPMs7BuDnVY4dfpJyDs3EIYYolkMcrWKYOtLPQvXqocqLvrs+e/6NlWr4QIy9Wl1/+uULjvg/Saa3GeSgw9bgRze0TrkGfgD8lDyxHMO35Xklr0MzcYhhawDMR29d8/Ygfbi+sEPQwztu/+JT/+atb32+G0WPH9RoozxyyAuHGLpRY82uwQ6/BHVdksYhDmLYupr7R2+5aZ9KZei/VlXdIwQ9q2pl3fr12d9+atGih7tVKg45sTUswLqld7vr4IdXjbbzq8jvAX4R9ZrMpYC8Cgg/8IMdZfNGR3141UcJSNp0SYBfgroUkFcB4Qd+AHyAn9LqXXpFSqx5xgD8PKq1meOSNA5xEAOwrS8Xh5zgSN8rL/GjBBA1uCTAL0FnGppXM8EP/GCHzw4/pdW79IqUWPOMcQf+SyRdJ+lASfHveS+VtLrVg/IufZo7O0rv5u7SVB3iIAb6VR5w553jDvx9JMV/bpD0Wkk3SdoD4KfZTTPZqJODDi5xOGjhEAN+eIEWP9J6etFR7sCf/nyvl/RpSfsD/DTbHRorMXg1VvzAD07AvE/A0rp7vlFbC/BfJ+lKSe+LG7apRx0bGzs3hHBO/aOPjIzkU4NZKIACKIACM1qB4eHhrYWLHfvk+GCXSzpJ0qOSdpV0gKRRScdLerzdE/IaPjsYdjDeOxiHUwaOkL36BH60I1t3fu8I/OlPNlfS1ZIWSHou5ZEBvlchOzR3hxhoaF55iR/40YgnLr0ihXV5xrgD/yxJ59U92MskPdXsYQG+VyE7FJBDDADGKy/xAz8Afp4lg9kcgO9VyA6wdYgBwHjlJX7gB8A3g3eecAC+VyE7wNYhBgDjlZf4gR8APw9hzeYAfK9CdoCtQwwAxisv8QM/AL4ZvPOEA/C9CtkBtg4xABivvMQP/AD4eQhrNgfgexWyA2wdYgAwXnmJH/gB8M3gnSccgO9VyA6wdYgBwHjlJX7gB8DPQ1izOQDfq5AdYOsQA4Dxykv8wA+AbwbvPOEAfK9CdoCtQwwAxisv8QM/AH4ewprNAfhehewAW4cYAIxXXuIHfgB8M3jnCQfgexWyA2wdYgAwXnmJH/gB8PMQ1mwOwPcqZAfYOsQAYLzyEj/wA+CbwTtPOADfq5AdYOsQA4Dxykv8wA+An4ewZnMAvlchO8DWIQYA45WX+IEfAN8M3nnCAfhehewAW4cYAIxXXuIHfgD8PIQ1mwPwvQrZAbYOMQAYr7zED/wA+GbwzhMOwPcqZAfYOsQAYLzyEj/wA+DnIazZHIDvVcgOsHWIAcB45SV+4AfAN4N3nnAAvlchO8DWIQYA45WX+IEfAD8PYc3mAHyvQnaArUMMAMYrL/EDPwC+GbzzhAPwvQrZAbYOMQAYr7zED/wA+HkIazYH4HsVsgNsHWIAMF55iR/4AfDN4J0nHIDvVcgOsHWIAcB45SV+4AfAz0NYszkA36uQHWDrEAOA8cpL/MAPgG8G7zzhAHyvQnaArUMMAMYrL/EDPwB+HsKazQH4XoXsAFuHGACMV17iB34AfDN45wkH4HsVsgNsHWIAMF55iR/4AfDzENZsDsD3KmQH2DrEAGC88hI/8APgm8E7TzgA36uQHWDrEAOA8cpL/MAPgJ+HsGZzAL5XITvA1iEGAOOVl/iBHwDfDN55wgH4XoXsAFuHGACMV17iB34A/DyENZsD8L0K2QG2DjEAGK+8xA/8APhm8M4TDsD3KmQH2DrEAGC88hI/8APg5yGs2RyA71XIDrB1iAHAeOUlfuAHwDeDd55wAL5XITvA1iEGAOOVl/iBHwA/D2HN5gB8r0J2gK1DDADGKy/xAz8Avhm884QD8L0K2QG2DjEAGK+8xA/8APh5CGs2B+B7FbIDbB1iADBeeYkf+AHwzeCdJxyA71XIDrB1iAHAeOUlfuAHwM9DWLM5AN+rkB1g6xADgPHKS/zAD4BvBu884QB8r0J2gK1DDADGKy/xAz8Afh7Cms0B+F6F7ABbhxgAjFde4gd+AHwzeOcJB+B7FbIDbB1iADBeeYkf+AHw8xDWbA7A9ypkB9g6xABgvPISP/AD4JvBe1o4syXdI+mDkm5uFSbA9ypkB9g6xABgvPISP/AD4PsC/3xJb5J0GcBPN8kBdMTg1VjxAz/qO4hDTrAAS+/rRUaGIpN7NHcvSR+R9LSkOwB+uuoOhUwMAAbANK9Z6oP6SO/oxUduDcD/Yg34p9UDf2xs7NwQwjn1MoyMjBRXhiugAAqgAArMOAWGh4e3Bi7m8sXxwS6XdJKkRyWdLOnFkj4vaZwdfmces3vYqJeDDi5xOGjhEAN+eO2s8aOz3p53tCPwpz/L7ZIOqHu4t0n6RrMH5k17XoXs0NwdYqCheeUlfuBHI4a49Iq8QG83zx340+Nnh9/OzbrfOyQvMXg1VvzAj/o24pATLMA6bO45h29NwE96RHb4NDQaWvNScWjuDjEAGK8+gR9JeCs8COAXlnDLC9DQvJoJfuCH6/GtQ246xADwSwBRg0sC/BJ0poAADIDxPmUAMF41ih8lgAjg90ZUgO/VTPADP1iAsQBL6f4uvSIl1jxj2OHnUa3NHJekcYiDGIBtfbk45AQ7Sq+8xI8SQMQOvzei0tC8mgl+4Ac7fHb4Kd3fpVekxJpnDDv8PKqxw09WzaGAHGJgB+O16MAP/HBdBCY31xwDAX4O0dpNATBezQQ/8MO1uTvkpkMMLMDaUaU7vwf43dFxs6tQQAAGwHCEnNJaHHqFQwwAPyVbio8B+MU13OIKFBDAB/gAP6W1OPQKhxgAfkq2FB8D8ItrCPBbaOjQTBxioKF5LQLxAz9cF+UlIGnTJWcE8BeeuvKPqtsOHapMu4eKfp1NVv/1lguPuqUsYQGMVzPBD/xwbe4OuekQAwuwsmi0+XUHHvjvWHbjH76Q/e6aejkzhW/fMr740jJkpoAADIDhSD+ltzj0CocYAH5KthQfM/DAXzA6sUzK3hwUVlYV7guVyZeqGo6V9PKKsrPXjB91V3EZN78CBQTwAT7AT+krDr3CIQaAn5ItxccMPPAPHb3+C0HhpdVs/fJbLzzm7ijZoaMTZwVlf6ksfPbmCxevLS4jwG+moUMzcYiBhua1CMQP/HBdlHebR9OvN/DAXzB6/d9L4RUVVT66ZvzI+2vAPzUoO6BSrX5qzUVHf73bAgMYr2aCH/jh2twdctMhBhZg3aZQ4+sNPPAXLlt1YpZpgbJwZ6bqXSFUdgvS/pmybTPNOv2W8Xf9vNtSU0AABsBwpJ/SVxx6hUMMAD8lW4qPGXjgz1+2eqdZ1eryLGjPzY82wsq144uvLy7hlleggAA+wAf4Kb3FoVc4xADwU7Kl+JiBB36U6O1Lr9thm1nbLA7S7grh34Mq/7R27Mh/LC5f4ytQQAAf4AP8lP7i0CscYgD4KdlSfMyMAH5xmTq7AgUE8AE+wE/pGg69wiEGgJ+SLcXHAPziGm5xBQoI4AN8gJ/SWhx6hUMMAD8lW4qPGTjgX3HFFdkzzzxTXBmugAIogAIoMKMU2GmnnfTe97534Lg4ZeLAPdj4+Hg2Ojra1+dyiCEa7BAHMfy+X6LFRi0cdHCJw0ELhxjwozfrqr6CsYxHdEhehxgoIC/Q4gd+NOp3Dr3CIQbqowwabnlNgF+CzhSQV3PHD/wAts0bHfXhVR8lIGnTJQF+CepSQF4FhB/4AfABfkqrd+kVKbHmGTNwwB8bGzt32bJl5+YRo1tzHGKIz+IQBzH8PqvQYqMWDjq4xOGghUMM+NEt+rS+zsABvzeycRcUQAEUQAEU2LoUAPhbl19EiwIogAIogAK5FAD4uWRjEgqgAAqgAApsXQoMOvDfKOmH0yz5pqS/7pFFsyXdI+mDkm7u0T2n32aOpC9I2k/S45JOkfTVHsfxEknXSTpQ0q8lLZW0uscxxNv9laRrJa2Q9Jke3z/W2GWS3iNpnaQTJH2vxzH0W4Opx3XIB4e6mG5/P/tEP/vjdA2Ok3S+pKjFOZKu6FF9fEzSx+vu9VJJv+3R/Xt+m0EH/tslLZL033qu7MYEflOt2fcD+PMUvyxI+rqkN0j6oqRX9ViHfSTFf26Q9FpJN0nao8cxbFvz4DeSHukD8N8h6WRJ8d/Rk7jgePUM02DqcR3ywaEuptvfzz7Rz/44pcHOtR61UFL8E6mxR93Z4/qIt4v96UJJh/Xh3j275aADf3FtBfeHtWSK4F/TA3X3kvQRSU9LuqNPO/ypx6zUdthnSdq/B8/e7Bavl/TpPsYQV/O/6gPwL5b009oJQ9TmJ5IOqp269NqOfmnQ6Dn7nQ8OddHvPtGv/jg9H46RdGhtc7S9pA9L+nKvC6O26Hi/pAf7cO+e3XLQgf/nkuI/q2q7qy9J2lXS+pIVjrvpCPzTDID/fO2I6nBJ3y/5uZtd/nWSrpT0PkkP9CmGfsHuKklrpy36vlV7mSe+3NPrn35pUP+cDvngUBf97hP96o/T8yG+zLdA0rtqp3/xFHA3SdUeFke8f3ypN57EDfTPIAL/ckknSXq0BvfpBt4l6ZASdlfT7xmT5sWSPh//bHiPgd/o2eNOJu6m4qr5zyTFRlfmT30MB0galXR8Cbo3e45GOvQLdhdJ+pmka2rB3i3pbbXX88v0odG1+6XB9Fj6kQ+NtOh1XdTHEAHXrz7RLO/K6o+t8vzEWp+OJ5Dx58eSDpb0ZA+L49bae5z6sQjv4WNKgwj86QLG19DjmzDim/XiG7fiG8jia0Rlrh5vlxSb2vSf2OC/0VNnN75/YDtJ35U0XFt4xNfP48sMvfqZK+nq2gr+uV7dtMl9+gW7+Nrk6bXXBuObpOIiML6O3I+ffmkw9awO+eBQF1EPhz7Rj/5Yn/d7197jE3fY8Q2VX6nt9Mvs0dNjiG8U/N99eF9NP+p/4IEfARffqf4Xkv5V0ockxSPVXv30eoc//bliIcVThri7j29YO68Pr1/HVXu87/Sfl0l6qlcG1D6l8J26++1Uez2/F2FMvUs/nnA8JulYST/qxY2n3SN+UqOfGkyF4pAPDnVRb3+/+kS/++OUDvFYPy6K46Yg9uhevsk5vrwU30j7lh7XZF9uN+g7/L6Iyk1RAAVQAAVQwE0BgO/mCPGgAAqgAAqgQAkKAPwSROWSKIACKIACKOCmAMB3c4R4UAAFUAAFUKAEBQB+CaJySRRAARRAARRwUwDguzlCPCiAAiiAAihQggIAvwRRuSQKoAAKoAAKuCkA8N0cIR4UQAEUQAEUKEEBgF+CqFwSBVAABVAABdwUAPhujhAPCqAACqAACpSgAMAvQVQuiQJmCpwt6czan5iOf9b4XyR9TlL8k6b8oAAKzBAFAP4MMZrHnNEKzJL0g5oC/1b7Qqn49/VfmNGq8PAoMMMUAPgzzHAed8YqEL80Jn716JCkvSQ9OGOV4MFRYIYqAPBnqPE89oxTIH5r4g8lbSvp/ZL++4xTgAdGgRmuAMCf4QnA488IBbavfSXv3ZKekBS/qvdPJT06I56eh0QBFNigAMAnEVBg8BX4pKR3S4rf/f2spJ/W/lk4+I/OE6IACkwpAPDJBRRAARRAARSYAQoA/BlgMo+IAiiAAiiAAgCfHEABFEABFECBGaAAwJ8BJvOIKIACKIACKADwyQEUQAEUQAEUmAEKAPwZYDKPiAIogAIogAIAnxxAARRAARRAgRmgAMCfASbziCiAAiiAAijw/wH+7Ig4wU4A1QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<VegaLite 2 object>\n",
       "\n",
       "If you see this message, it means the renderer has not been properly enabled\n",
       "for the frontend that you are using. For more information, see\n",
       "https://altair-viz.github.io/user_guide/troubleshooting.html\n"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "def pca_2d(paragraph_matrix, groups):\n",
    "    pca = PCA(n_components=2)\n",
    "    reduced_dims = pca.fit_transform(paragraph_matrix)\n",
    "    print(f\"2-component PCA, explains {sum(pca.explained_variance_):.2f}% of variance\")\n",
    "    df = pd.DataFrame(reduced_dims, columns=[\"x\", \"y\"])\n",
    "    df[\"group\"] = groups\n",
    "    return df\n",
    "\n",
    "example_2d = pca_2d(model.paragraph_matrix.data, [\"0\",\"1\",\"2\",\"3\"])\n",
    "alt.Chart(example_2d).mark_point().encode(x=\"x\", y=\"y\", color=\"group\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not much to see on such a tiny dataset without any labelled groups."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running this on some bigger data\n",
    "--------------------------------\n",
    "\n",
    "We'll use the BBC's dataset. The dataset was created by Derek Greene at UCD and all articles are copyright Auntie. I've munged it into a file per topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>tokens</th>\n",
       "      <th>group</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Claxton hunting first major medal  British hurdler Sarah Claxton is confident she can win her fi...</td>\n",
       "      <td>[claxton, hunting, first, major, medal, british, hurdler, sarah, claxton, is, confident, she, ca...</td>\n",
       "      <td>sport</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>O'Sullivan could run in Worlds  Sonia O'Sullivan has indicated that she would like to participat...</td>\n",
       "      <td>[could, run, in, worlds, sonia, has, indicated, that, she, would, like, to, participate, in, nex...</td>\n",
       "      <td>sport</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Greene sets sights on world title  Maurice Greene aims to wipe out the pain of losing his Olympi...</td>\n",
       "      <td>[greene, sets, sights, on, world, title, maurice, greene, aims, to, wipe, out, the, pain, of, lo...</td>\n",
       "      <td>sport</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>IAAF launches fight against drugs  The IAAF - athletics' world governing body - has met anti-dop...</td>\n",
       "      <td>[iaaf, launches, fight, against, drugs, the, iaaf, athletics, world, governing, body, has, met, ...</td>\n",
       "      <td>sport</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                  text  \\\n",
       "0  Claxton hunting first major medal  British hurdler Sarah Claxton is confident she can win her fi...   \n",
       "1  O'Sullivan could run in Worlds  Sonia O'Sullivan has indicated that she would like to participat...   \n",
       "2  Greene sets sights on world title  Maurice Greene aims to wipe out the pain of losing his Olympi...   \n",
       "3  IAAF launches fight against drugs  The IAAF - athletics' world governing body - has met anti-dop...   \n",
       "\n",
       "                                                                                                tokens  \\\n",
       "0  [claxton, hunting, first, major, medal, british, hurdler, sarah, claxton, is, confident, she, ca...   \n",
       "1  [could, run, in, worlds, sonia, has, indicated, that, she, would, like, to, participate, in, nex...   \n",
       "2  [greene, sets, sights, on, world, title, maurice, greene, aims, to, wipe, out, the, pain, of, lo...   \n",
       "3  [iaaf, launches, fight, against, drugs, the, iaaf, athletics, world, governing, body, has, met, ...   \n",
       "\n",
       "   group  \n",
       "0  sport  \n",
       "1  sport  \n",
       "2  sport  \n",
       "3  sport  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfs = []\n",
    "for document_set in (\"sport\",\n",
    "                     \"business\",\n",
    "                     \"politics\", \n",
    "                     \"tech\", \n",
    "                     \"entertainment\"):\n",
    "    df_ = pd.read_csv(f\"data/bbc/{document_set}.csv.bz2\", encoding=\"latin1\")\n",
    "    df_ = tokenize_text(df_)\n",
    "    df_[\"group\"] = document_set\n",
    "    dfs.append(df_)\n",
    "\n",
    "bbc_df = pd.concat(dfs)\n",
    "bbc_df[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset comprises 2225 documents and 19063 unique words\n"
     ]
    }
   ],
   "source": [
    "bbc_vocab = Vocab([tok for tokens in bbc_df.tokens for tok in tokens])\n",
    "\n",
    "bbc_df = clean_tokens(bbc_df, bbc_vocab)\n",
    "\n",
    "print(f\"Dataset comprises {len(bbc_df)} documents and {len(bbc_vocab.words)} unique words\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "bbc_noise = NoiseDistribution(bbc_vocab)\n",
    "bbc_examples = list(example_generator(bbc_df, context_size=5, noise=bbc_noise, n_negative_samples=5, vocab=bbc_vocab))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "bbc_dataset = NCEDataset(bbc_examples)\n",
    "bbc_dataloader = DataLoader(bbc_dataset, batch_size=1024, drop_last=True, shuffle=True)  # TODO could tolerate a larger batch size\n",
    "\n",
    "bbc_model = DistributedMemory(vec_dim=50,\n",
    "                              n_docs=len(bbc_df),\n",
    "                              n_words=len(bbc_vocab.words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epochs: 100%|██████████| 80/80 [36:14<00:00, 26.78s/it]\n"
     ]
    }
   ],
   "source": [
    "bbc_training_losses = train(bbc_model, bbc_dataloader, epochs=80, lr=1e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.vegalite.v2+json": {
       "$schema": "https://vega.github.io/schema/vega-lite/v2.6.0.json",
       "config": {
        "view": {
         "height": 300,
         "width": 400
        }
       },
       "data": {
        "name": "data-1dabe6fce07510d2b0d23042635cb568"
       },
       "datasets": {
        "data-1dabe6fce07510d2b0d23042635cb568": [
         {
          "epoch": 0,
          "training_loss": 2.642270294331616
         },
         {
          "epoch": 1,
          "training_loss": 2.1742215334258463
         },
         {
          "epoch": 2,
          "training_loss": 1.990165346749821
         },
         {
          "epoch": 3,
          "training_loss": 1.873253081452032
         },
         {
          "epoch": 4,
          "training_loss": 1.789763425596012
         },
         {
          "epoch": 5,
          "training_loss": 1.7244694537997987
         },
         {
          "epoch": 6,
          "training_loss": 1.670895414915144
         },
         {
          "epoch": 7,
          "training_loss": 1.6244636086943727
         },
         {
          "epoch": 8,
          "training_loss": 1.5836919797873645
         },
         {
          "epoch": 9,
          "training_loss": 1.5470218249729701
         },
         {
          "epoch": 10,
          "training_loss": 1.5139503951398483
         },
         {
          "epoch": 11,
          "training_loss": 1.4832230216968134
         },
         {
          "epoch": 12,
          "training_loss": 1.4547421681954993
         },
         {
          "epoch": 13,
          "training_loss": 1.4282586559745836
         },
         {
          "epoch": 14,
          "training_loss": 1.403701515079285
         },
         {
          "epoch": 15,
          "training_loss": 1.3798937556166087
         },
         {
          "epoch": 16,
          "training_loss": 1.3577345677784511
         },
         {
          "epoch": 17,
          "training_loss": 1.3366852766238384
         },
         {
          "epoch": 18,
          "training_loss": 1.3162777810363295
         },
         {
          "epoch": 19,
          "training_loss": 1.2970764010589315
         },
         {
          "epoch": 20,
          "training_loss": 1.2788146187800058
         },
         {
          "epoch": 21,
          "training_loss": 1.2611135737496133
         },
         {
          "epoch": 22,
          "training_loss": 1.2440896054972772
         },
         {
          "epoch": 23,
          "training_loss": 1.2277287255162779
         },
         {
          "epoch": 24,
          "training_loss": 1.211776340229911
         },
         {
          "epoch": 25,
          "training_loss": 1.1965603542624053
         },
         {
          "epoch": 26,
          "training_loss": 1.1817405841365365
         },
         {
          "epoch": 27,
          "training_loss": 1.1673207131972223
         },
         {
          "epoch": 28,
          "training_loss": 1.1533892758884785
         },
         {
          "epoch": 29,
          "training_loss": 1.1397850001080436
         },
         {
          "epoch": 30,
          "training_loss": 1.12681249217217
         },
         {
          "epoch": 31,
          "training_loss": 1.1138663835407043
         },
         {
          "epoch": 32,
          "training_loss": 1.1015954906896035
         },
         {
          "epoch": 33,
          "training_loss": 1.089359670544263
         },
         {
          "epoch": 34,
          "training_loss": 1.0776376641314962
         },
         {
          "epoch": 35,
          "training_loss": 1.0659556754627584
         },
         {
          "epoch": 36,
          "training_loss": 1.0547422724480955
         },
         {
          "epoch": 37,
          "training_loss": 1.0438089944561075
         },
         {
          "epoch": 38,
          "training_loss": 1.0329133888209088
         },
         {
          "epoch": 39,
          "training_loss": 1.0227227165832282
         },
         {
          "epoch": 40,
          "training_loss": 1.0123565251042383
         },
         {
          "epoch": 41,
          "training_loss": 1.0023637906364773
         },
         {
          "epoch": 42,
          "training_loss": 0.9926695086200785
         },
         {
          "epoch": 43,
          "training_loss": 0.982954163951163
         },
         {
          "epoch": 44,
          "training_loss": 0.9737152805239517
         },
         {
          "epoch": 45,
          "training_loss": 0.9645086004867317
         },
         {
          "epoch": 46,
          "training_loss": 0.9555849193786242
         },
         {
          "epoch": 47,
          "training_loss": 0.9466561066437952
         },
         {
          "epoch": 48,
          "training_loss": 0.9380779170101474
         },
         {
          "epoch": 49,
          "training_loss": 0.929785754813911
         },
         {
          "epoch": 50,
          "training_loss": 0.9214358320147354
         },
         {
          "epoch": 51,
          "training_loss": 0.9133923515770006
         },
         {
          "epoch": 52,
          "training_loss": 0.905320764328382
         },
         {
          "epoch": 53,
          "training_loss": 0.8976051105475574
         },
         {
          "epoch": 54,
          "training_loss": 0.8899229808623746
         },
         {
          "epoch": 55,
          "training_loss": 0.8823469078318673
         },
         {
          "epoch": 56,
          "training_loss": 0.8750101461173585
         },
         {
          "epoch": 57,
          "training_loss": 0.867823110456052
         },
         {
          "epoch": 58,
          "training_loss": 0.8607556018029681
         },
         {
          "epoch": 59,
          "training_loss": 0.8537966694891083
         },
         {
          "epoch": 60,
          "training_loss": 0.8470302788367182
         },
         {
          "epoch": 61,
          "training_loss": 0.8402785332306572
         },
         {
          "epoch": 62,
          "training_loss": 0.8336316384884142
         },
         {
          "epoch": 63,
          "training_loss": 0.8271638499283642
         },
         {
          "epoch": 64,
          "training_loss": 0.8209897966118332
         },
         {
          "epoch": 65,
          "training_loss": 0.8146745335981712
         },
         {
          "epoch": 66,
          "training_loss": 0.8085972012940401
         },
         {
          "epoch": 67,
          "training_loss": 0.8024606412982348
         },
         {
          "epoch": 68,
          "training_loss": 0.7965273064856203
         },
         {
          "epoch": 69,
          "training_loss": 0.7905283134916554
         },
         {
          "epoch": 70,
          "training_loss": 0.785075603212629
         },
         {
          "epoch": 71,
          "training_loss": 0.7794511792822654
         },
         {
          "epoch": 72,
          "training_loss": 0.7739284674573389
         },
         {
          "epoch": 73,
          "training_loss": 0.7684736896745907
         },
         {
          "epoch": 74,
          "training_loss": 0.7630798336141598
         },
         {
          "epoch": 75,
          "training_loss": 0.7578280769519924
         },
         {
          "epoch": 76,
          "training_loss": 0.7526835779966035
         },
         {
          "epoch": 77,
          "training_loss": 0.7476536778189381
         },
         {
          "epoch": 78,
          "training_loss": 0.7426093139263413
         },
         {
          "epoch": 79,
          "training_loss": 0.7375516610856382
         }
        ]
       },
       "encoding": {
        "x": {
         "field": "epoch",
         "type": "quantitative"
        },
        "y": {
         "field": "training_loss",
         "type": "quantitative"
        }
       },
       "mark": "bar"
      },
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcoAAAFZCAYAAAAYbm8xAAAgAElEQVR4Xu2dC7QlVZnff6dRQaODxmFGRWOw8UWiBhAHZxlHx7dRGJNppPER8ZFJ1Dhmoq52loZuUe8FE3FFx1FxfOEIikSlm4dRGJ+oKKBg0HFkBlTAxOf4QkVuZW2oC+eevufeXfVVnburz++s1avh9v7vs+v3fbX/99tVtWuEHwlIQAISkIAEphIYyUYCEpCABCQggekENEqzQwISkIAEJLAGAY3S9JCABCQgAQlolOaABCQgAQlIoB2BPa6i3LlzZ3XQQQe1o6FKAhKQgATmmsDmzZt388U9zigXFxerbdu2FXtcV1xxRbVaIErJTMcXi4T85BcjEFObf/3wK9ZQ2h6uRtmW3E06TzT5xQjE1Oaf/GIEYupp+adRxrg2VjsRNEa2QiA/+cUIxNTm33zy0yhjcW+s9kRrjEyjjCGTn/w6JBDraqjzn0YZi3tj9VATpfGB9iSQXwys/OQXIxBTDzX/NMpY3Burh5oojQ+0J4H8YmDlJ78YgZh6qPmnUcbi3lg91ERpfKA9CeQXAys/+cUIxNRDzT+NMhb3xuqhJkrjA+1JIL8YWPnJL0Ygph5q/mmUsbg3Vg81URofaE8C+cXAyk9+MQIx9VDzT6OMxb2xeqiJ0vhAexLILwZWfvKLEYiph5p/GmUs7o3VQ02Uxgfak0B+MbDyk1+MQEw91PzTKGNxb6weaqI0PtCeBPKLgZWf/GIEYuqh5p9GGYt7Y/VQE6XxgfYkkF8MrPzkFyMQUw81/zTKWNwbq4eaKI0PtCeB/GJg5Se/GIGYeqj5p1HG4t5YPdREaXygPQnkFwMrP/nFCMTUQ80/jTIW9yz1ES879W7LDbc/4+Crt59yyf7p/888ces1WR3MsNFQE3mGiNb8KvnFIiE/+cUIxNTT8k+jjHHNUj9p26nbYXTcysbVjl2LW7dndTDDRk5UMdjyk1+MQExt/vXDT6OMcc1Sa5RZmLIaORFkYZraSH7yixGIqYeafxplLO5Zao0yC1NWo6GeaFkHN4NG8otBlt988tMoY3HPUmuUWZiyGjlRZWGyooxhkp/8VhDQKHtKiPFuNcruIGuUMZbyk1+MQEw91PzTKGNxz1JrlFmYshoN9UTLOrgZNJJfDLL85pOfRhmLe5Zao8zClNXIiSoLk0uHMUzyk59Lrz3lwNRuNcruiGuUMZbyk1+MQEw91PyzoozFPUutUWZhymo01BMt6+Bm0Eh+Mcjym09+GmUs7llqjTILU1YjJ6osTC4dxjDJT34uvfaUAy69zgCsRhmDLD/5xQjE1EPNPyvKWNyz1FaUWZiyGg31RMs6uBk0kl8Msvzmk99GG+W+wHuARwE/BF4CfGAiFIcBF4797Dzg0dPCtbi4WG3btm2jj2vF8DTK2Mk1rnaiirGUn/xiBGLqoebfRhvKA4D05wzgfsCHgQMmQvFY4AjghTkh0ihzKE1vM9REjh11d2r5xVjKT34xAjH1tPzbaKMcP6oHAm8CHj5xqE8FjgfuDPy8NswzrShjCTFN7UQV4yo/+cUIxNTmXz/8SjHK+wNvBp4LXDFxqIcA6c9pwMHA6cDdgd8sLCxsH40mX18FW7ZsidHqWH3uxddwzkVXr+j1CYfuz+MPufk1lR1/o91JQAISkEAbAps3b97NF0swykcA24BjgWszDuwS4InT2rr0mkFwjSb+Riq/GIGY2vyTX4xATF3q0us9gXcATwKum3KIhwN3BNJNPA+tb/65F7C0WnuNsp9EifXandqJNMZSfvKLEYiph5p/G11RvhJ41QT6OwH7AJ8BDqxv7jkFOBS4EngBcP60cGmU85nIsaPuTj3UiaA7ArGe5Ce/GIGYutSKMnZUq6g1yhhSJyr5xQjE1Oaf/GIEYmqNMsYvpPY5yhC+FWIn0hhL+ckvRiCmHmr+bfTSa4y6FWXn/IaayJ2DaNmh/FqCq2Xyk1+MQExtRRnjF1JbUYbwWVF2hw+NKAZTfvPJz4oyFvcstUaZhSmrkRNVFqapjeQnvxiBmHqo+adRxuKepdYoszBlNRrqiZZ1cDNoJL8YZPnNJz+NMhb3LLVGmYUpq5ETVRYmK8oYJvnJbwUBjbKnhBjvVqPsDrJGGWMpP/nFCMTUQ80/jTIW9yy1RpmFKavRUE+0rIObQSP5xSDLbz75aZSxuGepNcosTFmNnKiyMLl0GMMkP/m59NpTDkztVqPsjrhGGWMpP/nFCMTUQ80/K8pY3LPUGmUWpqxGQz3Rsg5uBo3kF4Msv/nkp1HG4p6l1iizMGU1cqLKwuTSYQyT/OTn0mtPOeDS6wzAapQxyPKTX4xATD3U/LOijMU9S21FmYUpq9FQT7Ssg5tBI/nFIMtvPvlplLG4Z6k1yixMWY2cqLIwuXQYwyQ/+bn02lMOuPQ6A7AaZQyy/OQXIxBTDzX/rChjcc9SW1FmYcpqNNQTLevgZtBIfjHI8ptPfhplLO5Zao0yC1NWIyeqLEwuHcYwyU9+Lr32lAMuvc4ArEYZgyw/+cUIxNRDzT8ryljcs9RWlFmYshoN9UTLOrgZNJJfDLL85pOfRhmLe5Zao8zClNXIiSoLk0uHMUzyk59Lrz3lgEuvMwCrUcYgy09+MQIx9VDzz4oyFvcstRVlFqasRkM90bIObgaN5BeDLL/55KdRxuKepdYoszBlNXKiysLk0mEMk/zk59JrTzng0usMwGqUMcjyk1+MQEw91PyzoozFPUttRZmFKavRUE+0rIObQSP5xSDLbz75aZSxuGepNcosTFmNnKiyMLl0GMMkP/m59NpTDrj0OgOwGmUMsvzkFyMQUw81/6woY3HPUltRZmHKajTUEy3r4GbQSH4xyPKbT34aZSzuWWqNMgtTViMnqixMLh3GMMlPfi699pQDDZde+TXc+Kf+VP9j1+LW7bMe2+T3aUSxCMhPfjECMbX51w8/K8oY1yz16hXlpLTaoVGuj9OJYH1Ga7WQn/xiBGLqoeafRhmLe5Zao8zClNVoqCda1sHNoJH8YpDlN5/8NMpY3LPUGmUWpqxGTlRZmKY2kp/8YgRi6qHmn0YZi3uWWqPMwpTVaKgnWtbBzaCR/GKQ5Tef/DTKWNyz1BplFqasRk5UWZisKGOY5Ce/FQQ0yp4SYrxbjbI7yBpljKX85BcjEFMPNf80yljcs9QaZRamrEZDPdGyDm4GjeQXgyy/+eSnUcbinqXWKLMwZTVyosrC5NJhDJP85OfSa085MLVbjbI74hpljKX85BcjEFMPNf+sKGNxz1JrlFmYshoN9UTLOrgZNJJfDLL85pOfRhmLe5Zao8zClNXIiSoLk0uHMUzyk59Lrz3lgEuvMwCrUcYgy09+MQIx9VDzz4oyFvcstRVlFqasRkM90bIObgaN5BeDLL/55KdRxuKepdYoszBlNXKiysLk0mEMk/zkV9TS677Ae4BHAT8EXgJ8YCJGycxPAp4FfBd4NnDBtDguLi5W27ZtK+oXAI2yu7NOo4yxlJ/8YgRi6qHm30YbygOA9OcM4H7Ah4EDJkLxFODFQPr7YOAtwL01yljCTlMPNZH7odG8V/k1ZzaukJ/8YgRi6mn5t9FGOX5UDwTeBDx84lBfB3wVeHf980uBxwHXrobEirKfRIn12p3aiTTGUn7yixGIqYeaf6UY5f2BNwPPBa6YCMVbgZ3Arvrn5wPPB76uUcaSdjX1UBO5exLtepRfO27LKvnJL0Ygpi65onwEsA04dkqVeCJwOfCuGsFlwGPS9cqFhYXto9HouEk0W7ZsidHqWH3uxddwzkVXr9nr4ffdj8Pv+9sr2hzwu7fveCR2JwEJSEACaxHYvHnzbgXkRleU9wTeATwJuG7K4J8MvAw4EjgMWKyvVa7afLhLr5OHU+3Ytbh1+6xT2t/oY8TlJ78YgZja/OuH30Yb5SuBV00c2p2AfYDPAAcCy3e9porzGuCZwBen4dAo+0mUWK/dqZ0IYizlJ78YgZh6qPm30UYZo76KWqOMIR1qIseOuju1/GIs5Se/GIGYuuRrlLEjm1BrlDGcTlTyixGIqc0/+cUIxNQaZYxfSJ234cDkV3iNcjXoTqShVER+8osRiKmHmn8uvcbinqXWKLMwZTUa6omWdXAzaCS/GGT5zSc/jTIW9yy1RpmFKauRE1UWpqmN5Ce/GIGYeqj5p1HG4p6l1iizMGU1GuqJlnVwM2gkvxhk+c0nP40yFvcstUaZhSmrkRNVFiYryhgm+clvBQGNsqeEGO9Wo+wOskYZYyk/+cUIxNRDzT+NMhb3LLVGmYUpq9FQT7Ssg5tBI/nFIMtvPvlplLG4Z6k1yixMWY2cqLIwuXQYwyQ/+bn02lMOTO22lVFW1d+NRpu+sdxpxdKXZrH3q0YUyw75yS9GIKY2//rhZ0UZ45qlbmWUu/U8mw0IPNGyQmrFEcMkP/n1RCDWrTvzxPiF1BplCN8KsUYeYyk/+cUIxNRDzT8ryljcs9QaZRamrEZDPdGyDm4GjeQXgyy/+eSnUcbinqXWKLMwZTVyosrC5NJmDJP85LeCgEbZU0KMd6tRdgdZo4yxlJ/8YgRi6qHmn0YZi3uWWqPMwpTVaKgnWtbBzaCR/GKQ5Tef/DTKWNyz1BplFqasRk5UWZhcOoxhkp/8XHrtKQemdqtRdkdco4yxlJ/8YgRi6qHmnxVlLO5Z6m6Mkm8BV93yhdX5fWxAMNREzgrEDBrJLwZZfvKLEYipfY4yxi+k7sgoJ8bQzwYETlShUCM/+cUIxNTmXz/8rChjXLPUGmUWpqxGTgRZmLzGFsMkP/mtIKBR9pQQ491qlN1B1ihjLOUnvxiBmHqo+adRxuKepdYoszBlNRrqiZZ1cDNoJL8YZPnNJz+NMhb3LLVGmYUpq5ETVRYmlw5jmOQnP5dee8qBqd1qlN0R1yhjLOUnvxiBmHqo+WdFGYt7lrofo0yPi4zSIyP1Z+m8Lh4XGWoiZwViBo3kF4MsP/nFCMTUXT0e8kDgUiAZ7IuB+wCvAb4TG1536sXFxWrbtm1F/QLQk1FOQOvmcREnqlguyk9+MQIxtfnXD7+mhnIh8BDgj4H3A38LfBt4XGx43ak1yhhLTzT5xQjE1Oaf/GIEYuquKspfA7cB3gXcGTgGuAa4Q2x43ak1yhhLJyr5xQjE1Oaf/GIEYuqujPLHwIOB84H/Cfx34FfA3rHhdafWKGMsnajkFyMQU5t/8osRiKm7MspTgKcD1wEHAVcCFwOHxIbXnXpujbKqXn3bffY6fpzk6TuOSisAjT5OVI1w7dZYfvKLEYipzb9++DW9RpmWXR9fX5tM1yfT53Dg87HhdaeeW6PcDWG7m3s80WK5KD/5xQjE1OZfP/yaGqV3vbaIw2zuep0cmEbZIlRhiRNVDKH85BcjEFN3tfTqXa8t4qBRtoA2ReJEGmMpP/nFCMTUQ82/phWld722yBONsgU0jbI7aGM9DXWi6gVGi07l1wLaHpB/TY3Su15b5MkGGeXCpqXbLYwP98wTj/zpesN3IliP0Nr/Lj/5xQjE1OZfP/yaGqV3vbaIw8YY5eRA865ZeqK1CPAe8Btz7Ki7U5t/MZby64dfU6P0rtcWcdAoW0Bz6bU7aBp5Zyw1ohjKofJrapSJ0u2ABwEV8JX6mcoYvQ7VPh4yDaYVZYdpNrWroU4Es2CT8x3yy6E0vY38+uHX1Ch/H/gQ8Dv1cNJm6EcAl8SG151ao9Qou8um5j05UTVnNq6Qn/xiBGLqLh8PuRY4FdgEPKPe5/VhseF1p9Yop7L82mjE15b/taqqy1Z7LZcTVSwX5Se/GIGY2vzrh1/TivJnwF2B5bsn71i/PcRN0deITxnXKCcHuPpSrCdaPydarNfu1MY3xlJ+88mvqVGmbeveCryjfifl84DnAPeN4etObUWZy1KjzCXVpJ0TaRNau7eVn/xiBGLqrpZej61N8uYVPOBZwHtiw+tOrVHmstQoc0k1aedE34SWRhmjJb9Z8WtaUaZx/V79ouYl4Bzgoq4HG+lPo8ylV10Fm9LbX+rP0ifSNUsn+lx+q7eTn/xiBGJq868ffrlGmd4Ystbn3NjwulNrlG1Z3lRheqK15XeTTn7yixGIqc2/fvjlGmV6ZnKtT24/q/XxUODdwOuBt6zS4DAgbca+/DkPePS0wWiUbRNFo2xLblznRBWjKD/5xQjE1NFrlOs9/vGZlu+lTDv9nAT8CEjPZK5mlI+tn9V8YQ4CjTKH0iptqurk0Wh08p/90UEXvv7Dlz8ktdi5ePQXW/bWm8yJNIZWfvKLEYiph5p/kUpwkliqOtv29wrg+1OM8qnA8cCdgZ8DyTDPtKKMJez66rydfNbvp9sWQz3RuqXQvjf5tWeXlPKbT35tjW01Wn0Z5SFA+nMacDBwOnB34DcLCwvbR6PRcZOD2bJlSyyaHavPvfgazrno6o577be7Jxy6P48/5G79fom9S0ACEiiMwObNm3fzxSEY5STGtF3eE4G0Q9BuH5deu8o6K8o2JK042lC7RSM/+cUIxNTRa5Q5395XRXk4kHYASjfxpBt/0jOb9wLS4ykaZU5k2rX5Rb3UXaurN6+25V27rturnEjbs0tK+ckvRiCmHmr+bXRFmW4S+vQE+v2AWwHpBqEDgQOA9B7MQ4H03N8LgPOnhcuKMpbI09VlVJhDPdH6ikrTfuXXlNjK9vKbT34bbZQx6laUnfPTKGNInUjlFyMQU5t//fDr0ijTDTbpEY8N/VhR9oY/PcKT/tSf6pSNWIp1IojFV37yixGIqYeaf02NMl23Wu3aYHqryAXAi4FvxVDG1BpljF++emOWYod6ouVz7bel/GJ85Tef/Joa5TuBR9R7vCbtE+pdc34APAb4u/pnMZoBtUYZgNdEOqpefcPP9n31uOScNz7xV026aNPWiaoNtVs08pNfjEBMPdT8a2qUXwCOBL5b40oP2n0ASDflpJtwrgB+K4YyptYoY/zaq2dTYQ71RGvPtVul/GI85Tef/JoaZbpG9d+A99W4jql3zUmPb/x5fUfq/jGUMbVGGePXXq1RJnZOpO0zSH4xdvLrj19To3xjvYXc+Ij+CnhuXWWmjc1PjA+3fQ8aZXt2MWW1q4Kd432ctbj1bbE+d1drRDGi8pNfjEBMPdT8a2qUqX3aFefBNa70Lsqz6xt80jOP34xhjKs1yjjDbnrop8Ic6onWDdN4L/KLMZTffPJrapTLlNJbPzaNIftlDF93ao2yO5axnjTKGL9+1E70Ma7ym09+TY3yWOCE+sadcWJN+4nRXkOtUfaGtmHH1Q9glN4IU3+q07p47tKJqmEYJprLT34xAjH1UPOvqcGlx0DSpuSfSG/vGEO2GMPXnVqj7I5ltz11U2EO9UTrlmX73uTXnl1Sym8++TU1yquABwL/GMPVn1qj7I9tqOeK8xgtpY3tb/7sWjxmoWmfTlRNia1sLz/5xQjE1EPNv6ZG+SLgMOCDwPjD5efG8HWn1ii7Y9lvT+0qzKGeaP2yzO9dfvmsVmspv/nk19Qo06u0Vvs07SdGew21Rtkb2q47fvvSqHr7eKdnL2xNG1qs+XGiWo+Q/GKE5Ce/+Iub0/Z149cml5mmV2IV8dEoiwhDi0HkVZgaZQu0YxL5yS9GIKYeav7lVoKPB9Lyavp7tY9Lr2vkz5O2nbodRsfFUmzPVlfwmp/vfd2KvWM/sePY3R47GuqJVkr05BeLhPzmk1+uUaYl19TWpdcWeaJRtoDG6hWmE1Ublrdo5Ce/GIGYeqj5l2uUadPztLya/l7t49KrFWXsDNpNrVF2DPTG7oY6UfXBok2f8mtDbfi/qOUa5SQdd+ZpkC9WlA1gLTetuIHR+PXwajFtWOBE1YLlmER+8osRiKmHmn9NjdKdeVrkiUbZAtruFebxP9v7Lq86+en3uP557/32rdM/f2LHI1e7sayLL2vdx1AngtYH3LFQfjGg8uuHX1OjdGeeFnHQKFtAW1eSd5fsut103MCJKgZUfvKLEYipp+VfU6N0Z54WcdAoW0BbRzKqeMMNt7rhDePNzn7N01J+bujHiT6GX37yixGIqbsySnfmaREHjbIFtMaSMipMJ/rGgVshkJ/8YgRi6q6M0sdDWsRBo2wBraGkojp102j0vnHZzoWjdzXsJtzciT6GUH7yixGIqbsySnfmaREHjbIFtLBkYypMJ/pY4OQnvxiBmDpqlO7ME+CvUQbgtZdeAHz2Fnn1iy7eh7necJzo1yO09r/LT34xAjF11CjdmSfAX6MMwOtMOpsK04k+FjD5yS9GIKaOGuVaO/PcHTgtNrzu1G6K3h3LPayn7wH/b6zC/GAfFaYTfSxr5Ce/GIGYOmqUy9+eHif5QyCZY/rvfYBXAvvHhtedWqPsjuUe3VNVnbxU8bbxYzz7xK1fih6zE32MoPzkFyMQU3dllG8B/mRsKDcA7wSeFxted2qNsjuW89VTN0uzTvSxrJGf/GIEYuqujPInwBHA3wAH1ab5TeBNseF1p9You2M5Zz1dMhpx8fIxV1X1nTZLs070sayRn/xiBGLqrozyOuC2wK+B+wO/Av4PsG9seN2pNcruWM51TxUnXv/rTSeMM/joSUf9cD0mTvTrEVr73+UnvxiBmLoro/wU8PD6lVsProd0NbA5Nrzu1BpldyztaZxA3tKsE30sa+QnvxiBmLoro3wo8HngHsBL6pt5/gL4Smx43ak1yu5Y2tMYgdHoQqrqwlt+Uv1gtaVZJ/pY1shPfjECMXVXRvkzYD8gLcEW+dEoiwzLHjeoaZuyO9HHQi0/+cUIxNRdGeUO4J7AmcAvxoZ0bmx43ak1yu5Y2lMTAjctzTrRN2G2e1v5yS9GIKbuyijdFL1FHNyZpwW0wUmqvx+NNl1xn7vd4THfuOanH6uqpQva3DXb92FrRDHC8ptPfk3fR/lI4PpVUH0mhq87tRVldyztKUTgjKrig+M9nHXC0Ru+g5UTfSimyG8++TU1ym3A4hiqvYCXTvwsRjKo1iiDAJX3Q2BUvfqGn+376vHOz3njE9PjVTP9ONHHcMtvPvk1McqnA6cAzxhDdSDwcmDvGL7u1BpldyztqU8CeY+bdD0CJ/oYUfnNJ78mRnkJ8K9WwXQ+8KgYvu7UGmV3LO2pVwLfHDFKu1rd+KlY+sIsrmk60cdiKr/55NfEKBOhdGI/egxVul55LbAUw9edWqPsjqU9zZTAx6mWPjb+jbtOOObErkfgRB8jKr/55NfUKGOUZqDWKGcA2a+YBYG3s3TDySuM88SnjW140G4ITvTtuC2r5Def/DTKWNyz1D4ekoXJRmsQqCpe9eB9vpaeY775s2PHjsYrOU70sTST33zy0yhjcc9Sa5RZmGzUiEC7m4Gc6BtB3q2x/OaTn0YZi3uWWqPMwmSjZgS+N4LvLUsqqtNzbgZyom8GebK1/OaTXwlGmTZafzfweiC9GHryk8Z4EvAs4LvAs4ELpoXLa5SxRFY9TAIVXD666ZV39ae63E3bu4+lRhljOlR+G22Ut6lN8EfAd6YY5VOAFwPp74PrNvfWKGMJq3oPJzDlbSdDnahKiZb8YpEYKr+NNspl6q8Avj/FKF8HfLWuOlP7S4HH1Y+l7BY1K8pYIqveYwl8vKL6+BEPucfimRd+O+2wxVmLW1e8mLqEIx/qRFoCuzQG+cUi0dWm6LFRTFevZZRvBXYCu2p52uDg+cDXV+tOo+wrRPa7hxHYSVWl8+rmz64Ttq54HGUjjteJPkZdfv3wG0JFmR66vhx4V43gMuAx6XrlwsLC9tFodNwkmi1btsRodaw+9+JrOOeiqzvu1e4k0B2BBx1wJx74z++0osMHH3jn7r7AniQwEAKbN2/ezReHYJRPBl4GHAkcVm/Anq5VrvqxohxINjrM0gm8H6r3r6g4F7d+qO9BWxHFCMuvH34bbZQPAz49cWj7AbcC0qu70qbry3e9HgtcAzwT+KJGGUsI1RJoQqCC00dL1QdWGOeJW1e8RqxJf9PaOtHHKMqvH34bbZSxo1pFbUXZOVI7lMDuBKrRBxktrTDKXYtbV1SgbbA50behdotGfv3w0yhjXLPUbjiQhclGgyZQnVnBR8YP4azFre9oekhO9E2JrWwvv374aZQxrllqjTILk432LALnVBXnrDDOE45+43qH6ES/HqG1/11+/fDTKGNcs9QaZRYmG+3RBEafHI2qTy4fYlVVlTsHdR9wjTLGtPTnKGNHN6b2GmVnKO1IAv0RqPgym0gvg7/pU1XfSsbpRB9DLr9++FlRxrhmqa0oszDZaI4JjEZcUVVccb+7/9Zjv/6dn/xvqD6Xs8n7rJFpRDHiQ+WnUcbinqXWKLMw2UgC4wR+NBrxw7Gl2veWYJxDnehLSa2h8tMoZ5BBGuUMIPsVc0VgVFV/ubTX6M3jB33Wa49Oe0L3+hnqRN8rlAadD5WfRtkgyG2bapRtyamTQDaBM5ZYOmO89dmLx5yarc5sONSJPvPwem82VH4aZe+pARrlDCD7FRIYI1DBZ0cVn735R6Pqui6Wboc60ZeSHEPlp1HOIIM0yhlA9isksAaBCq4cwZW3NKk+2cY4hzrRl5IcQ+WnUc4ggzTKGUD2KyTQiEC1NGJ0Q5JUjEZVVb17tGnpr8a72LVwzOcmuxzqRN8ITY+Nh8pPo+wxKZa71ihnANmvkECnBKrzq4rzxrs864Strx3qRN8pmkBnQ+WnUQaCnivVKHNJ2U4CpRKoroJNVx5419v/wTev/emnK/jUz/e+7tXjo/3EjmN/udGjH6oRbTS35e93Z54NjIRGuYHw/WoJzIbAhry/06XhboOrUXbLs1FvGmUjXDaWwPAJjPhylbbpqz8jqqva3DzUFIQVZVNiK9trlDF+IbVGGcKnWAJ7AoHfANfXB7IX6UXYFX89fmA7Tzh6xdtW2hy0RtmG2i0ajTLGL6TWKEP4FEtgDghUl8LoK7ccaJpnZzIAAA+KSURBVPX3bSpQjTKWKhpljF9IrVGG8CmWwDwSSI+u3Pj4CrAJ+EhV8cFxEGedcPRpk2A0yliqaJQxfiG1RhnCp1gCEtidwDeAv61/fNuq4qsjNp20/ZgHXLX9fZfdM/181wlHfas0cEM1ch8PmUEmaZQzgOxXSEACNxNIj6+MGN38omxYWmqzlNs1Uo2ya6It+/PFzS3BKZOABPZkAr8A0p/02Zuq+uiI6vTxA955wjEf6BuARtk34cz+NcpMUDaTgAQkcAuBq0Zw1Y3/O+L2S1X1pdENo7eOA9r1uqMvjgLTKKMEO9JrlB2BtBsJSEACNYEKLhvBpTf9b3U7qtE/cKtNr19hpK856ur1gGmU6xGa0b9rlDMC7ddIQAISuIXAF4DP1/+7F1X149vus9fx44BO33HUrzXKQlJGoywkEA5DAhKQwM0Eqh3pZiKNspCU0CgLCYTDkIAEJKBRlpkDGmWZcXFUEpDAPBOwoiwq+hplUeFwMBKQgATSDUAuvZaUBxplSdFwLBKQgAQSAY2yqDzQKIsKh4ORgAQkoFGWlgMaZWkRcTwSkIAErCiLygGNsqhwOBgJSEACVpSl5YBGWVpEHI8EJCABK8qickCjLCocDkYCEpCAFWVpOaBRlhYRxyMBCUjAirKoHNAoiwqHg5GABCRgRVlaDmiUpUXE8UhAAhKwoiwqBzTKosLhYCQgAQlYUZaWAxplaRFxPBKQgASsKIvKAY2yqHA4GAlIQAJWlKXlgEZZWkQcjwQkIAEryqJyQKMsKhwORgISkIAVZWk5oFGWFhHHIwEJSMCKsqgc0CiLCoeDkYAEJGBFWVoOaJSlRcTxSEACErCiLCoHNMqiwuFgJCABCVhRBnNgBJwEPAv4LvBs4IKJPg8DLhz72XnAo6d9r0YZjIhyCUhAAp0TsKKMIH0K8GIg/X0w8Bbg3hMdPhY4AnhhzhdplDmUbCMBCUhglgQ0ygjt1wFfBd5dd3Ip8Djg2rFOnwocD9wZ+HltmGdaUUawq5WABCQwSwIaZYT2W4GdwK66k/OB5wNfH+v0ECD9Oa2uOk8H7g78ZmFhYftoNDpucgBbtmyJjKlz7bkXX8M5F13deb92KAEJSGAIBJ5w6P48/pC7DWGobN68OV0SXPHZ7QczPpITgcuBd9XfexnwmPp65bShXAI8caLqvLmtS68zjqBfJwEJSGBdAlaU6yJao8GTgZcBRwLppp3FumoclxwO3BFIN/E8FHgPcC9gabV+NcpIONRKQAIS6IOARhmhunzX67HANcAzgS8CdwE+AxwIHACcAhwKXAm8AEhLtKt+NMpIONRKQAIS6IOARtkH1dZ9apSt0SmUgAQk0BMBjbInsO261SjbcVMlAQlIoD8CGmV/bFv0rFG2gKZEAhKQQK8ENMpe8TbtXKNsSsz2EpCABPomoFH2TbhR/xplI1w2loAEJDADAhrlDCDnf4VGmc/KlhKQgARmQ0CjnA3nzG/RKDNB2UwCEpDAzAholDNDnfNFGmUOJdtIQAISmCUBjXKWtNf9Lo1yXUQ2kIAEJDBjAhrljIGv/XUaZVHhcDASkIAEfHFzaTmgUZYWEccjAQlIwIqyqBzQKIsKh4ORgAQkYEVZWg5olKVFxPFIQAISsKIsKgc0yqLC4WAkIAEJWFGWlgMaZWkRcTwSkIAErCiLygGNsqhwOBgJSEACVpSl5YBGWVpEHI8EJCABK8qickCjLCocDkYCEpCAFWVpOaBRlhYRxyMBCUjAirKoHNAoiwqHg5GABCRgRVlaDmiUpUXE8UhAAhKwoiwqBzTKosLhYCQgAQlYUZaWAxplaRFxPBKQgASsKIvKAY2yqHA4GAlIQAJWlKXlgEZZWkQcjwQkIAEryqJyQKMsKhwORgISkIAVZWk5oFGWFhHHIwEJSMCKsqgc0CiLCoeDkYAEJGBFWVoOaJSlRcTxSEACErCiLCoHNMqiwuFgJCABCVhRlpYDGmVpEXE8EpCABKwoi8oBjbKocDgYCUhAAlaUpeWARllaRByPBCQgASvKonJAoywqHA5GAhKQgBVlaTmgUZYWEccjAQlIwIqyqBzQKIsKh4ORgAQkYEVZWg5olKVFxPFIQAISsKIsKgc0yqLC4WAkIAEJWFGWlgMaZWkRcTwSkIAErCiLygGNsqhwOBgJSEACVpSl5YBGWVpEHI8EJCABK8qickCjLCocDkYCEpCAFWVpOaBRlhYRxyMBCUjAirKoHNAoiwqHg5GABCRgRVlaDmiUpUXE8UhAAhKwoiwqBzTKosLhYCQgAQlYUZaWAxplaRFxPBKQgASsKCM5MAJOAp4FfBd4NnDBRIc5bW6WaJSRcKiVgAQk0AcBjTJC9SnAi4H098HAW4B7T3SY00ajjERBrQQkIIFeCWiUEbyvA74KvLvu5FLgccC1Y53mtNEoI1FQKwEJSKBXAhplBO9bgZ3ArrqT84HnA18f63Rqm4WFhe2j0ei48QHc+ta35vrrr4+MSa0EJCABCcwhgf3224/nPOc56XLfis9uP5gxmxOBy4F31d97GfCY+nrl8lBy2hRdUY4zLfEaquPrLuuNb4yl/OQXIxBTT8u/jTbKJwMvA44EDgMW62uV40eb00ajjOWH/OTXEYFYNxql/GIEYupSjXL5jtZjgWuAZwJfBO4CfAY4EJjWZlUinmj9JEqs1+7UxjfGUn7yixGIqYeafxtdUcaor6IeaiA6B9GyQ/m1BFfL5Ce/GIGY2vzrh98eZ5TpBp+Xv/zl22O4+lM7vhhb+ckvRiCmNv/mk98eZ5SxMKqWgAQkIAEJrCSgUZoREpCABCQggTUIaJSmhwQkIAEJSECj5KH17j+vr7fJS0ga7SHbQRbN+vtyh1wCm2lj3Rd4D/Ao4IfAS4APbEDspo3vd4FTgIfVu0n9GfCRgsa3PO7b1Zt4pM080uYepeRieiTswjG45wGPLmh8aWj/HngNkBimzU3eWND4XgEcP5GcdwR+krGHdu78EGl3a+CdwBHA9+rtStMGM7POv72Ak4F/B3wLeAbw5SbjmIeK8jZ10vwI+M6YUTbaQzaSLbV21t+XM+RS2Ewb6wOA9OcM4H7Ah4ED6r2B19sjOOf4o23S/sT/DPgY8CDgr4F7FTS+5eNLE/3h9XmQjLKUXHxsPYm+cCIQpYwvPaaWYpue5f55HdsvFMRvHFs6P06on0kvhd8f16b0NOA+wAc36Pw4qv6FJ43jCcB/Bn6/SRznwSiXkyn99vX9MaNstIdsdEYFZv19TYa80WxyxvpA4E3Awwtkuamuel9Z4PjSBPUi4Gf1s8nJKEvJxafWFdGdayNKhnlmQeN7OvBv6l8y9qkn2DTZl8Jv/LxJhv4fgH8oaHyPBNIqyzH1M/FpO9KHbMD40krA1cDba2D/F7hbvcHNenuN3yiZZ6PM2Wc2ZwLPbTPr78sdV2o3aZSljfX+wJuB5wJXAKWN75fAPwJ/BHyusPGlKjcZ5UvHjLIUfocA6c9p9Y5cpwN3B/4iYw/oJvndtm1a6n8SkCqjtJKRVjTuAfxlIeNbPq40xrRknVZZ0qeU+CZ/SczS0uuvgH8LnL0B40tGnarJrXDjZbi0/Ps7dQW+3l7je7RRvgH40/q3iHTirWYGjfaQbXumjelm/X1NhjxplCWN9RHANiDt3rT8VpmSxpc4p4oyVbyp2viXwKsy9jBuEp+2bdMEf4f6OlHaHjLtdpUqytL4LR/fJcATgf9SCL8/qY07rRSkz8XA4+tr5evtUd02Zm10yXxS5bb8MolS4vu8+heg9AvHXWuTTJcoZn1+pGul6RfGlFv/q76nIF0iyeY0zxVloz1k22TvhGbW39dkyJNGWcpY7wm8o/6t/rqxAyplfOm6397AZ4HNtRGlyiMtOa23h3GT+LRt+zdA+kVj/JNeOnDbQsaX+KWbT9JNPOk3/XTjVprA0nJnCfwOqq+Pp2ot3bj1obqyLGV8Ka7pJqOvTLzHt5TzI80raYkzGWW63ntBPc6Uk7OMbyqWDgU+Xi8Dp+uUqbrN5jQPRpnuSPz0xGSxH/CD+uaGyX1m205K6+ka7Vm7Xmcd/XspbKYdTvpNPv32Of65U73MeVJdZY7vEdwRluxu0kSa7upL1WS6WSyNNb18vMRYj1eUpYwv/VKR7hpOk9iVwAuA9Kq9UsaXEiFN8mlST7+opfGN3zU8q7ljrYRMlyVSzv3BWKNS+P028F7gX9fn7Gvr+wxmPb7fqq99/x7wpXoZNt39mj2OeTDK7FnPhhKQgAQkIIFJAhqlOSEBCUhAAhJYg4BGaXpIQAISkIAENEpzQAISkIAEJNCOgBVlO26qJCABCUhgTgholHMSaA9TAhKQgATaEdAo23FTJQEJSEACc0JAo5yTQHuYEpCABCTQjoBG2Y6bKgns6QSWN6NIu5icu6cfrMcngbUIaJTmhwQksBoBjdK8kEBNQKM0FSQwPAJpA+z/Wu/Zml4U/Z/q9+ulPVPTNmHp/Xv/BPhz4F314aWX1b68fvtFerVQetNEerdi+qTXW6Wt2v5pvR/ms+vN3dPWj+kVRc+p+0ttlvsbHjVHLIGWBDTKluCUSWCDCPxhvYn4o4BvAl8EFuo9LJOxvb9+L2Hagza96SK9tSG91Pfz9Z6lb6v3V037XqYNyNPbHNJm1f+x3sc0bbCdNqJP74VM/b2v3uM0vXXhX9Sbg2/Qofu1EtgYAhrlxnD3WyXQlkCqGFNlOP45A0ivlkvGdmRtcukdfOnVQskQk2HuANKG8j8GjgZOravQZLjHA/sCPxnrdHnp9Sn1OwXTmyBSu/S2lF+3Hbw6CQyRgEY5xKg55nkmsPxGlWnGtmyUT68rx8OAdENOerNJWlpNbzlZNtH0mqtkotvr112ll08vfyavUaZ3gqbKNb2iK72o2o8E5oaARjk3ofZA9xAC6XVKnwBeVFeO6V2Yqcq8tK4oU6WYXjic3u2YqsW09JpeB5auR74UOLmuJtOS672BBwOfAp4PnAVcCKTrnun1V6lCXb7rVaPcQxLIw2hOQKNszkyFBDaaQHo/YjLK29fLoskYU+WYjO2N9dLqb+obdNI1xvRJ705MZrc/8GXgT4GL6n9LNwelm3vS0mx6H+RzgftqlBsdZr+/FAIaZSmRcBwSiBFYXipNS6kfjXWlWgISGCegUZoPEtgzCPjc454RR4+iQAIaZYFBcUgSaEFAo2wBTYkEcgholDmUbCMBCUhAAnNLQKOc29B74BKQgAQkkENAo8yhZBsJSEACEphbAhrl3IbeA5eABCQggRwCGmUOJdtIQAISkMDcEvj/bAzS/QHkyUQAAAAASUVORK5CYII=",
      "text/plain": [
       "<VegaLite 2 object>\n",
       "\n",
       "If you see this message, it means the renderer has not been properly enabled\n",
       "for the frontend that you are using. For more information, see\n",
       "https://altair-viz.github.io/user_guide/troubleshooting.html\n"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alt.Chart(pd.DataFrame(enumerate(bbc_training_losses), columns=[\"epoch\", \"training_loss\"])).mark_bar().encode(x=\"epoch\", y=\"training_loss\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the reduced dimensionality paragraph vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bbc_2d = pca_2d(bbc_model.paragraph_matrix.data, bbc_df.group.to_numpy())\n",
    "chart = alt.Chart(bbc_2d).mark_point().encode(x=\"x\", y=\"y\", color=\"group\")\n",
    "# Uncomment to print chart inline, but beware it will inflate the notebook size\n",
    "# chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2-component PCA, explains 2.65% of variance`\n",
    "\n",
    "![](./img/bbc_pca_all_topics.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These results aren't great, but we can see the beginnings of separation. If we look at just two topics it becomes more obvious."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chart = alt.Chart(bbc_2d[bbc_2d[\"group\"].isin([\"sport\", \"business\"])]).mark_point().encode(x=\"x\", y=\"y\", color=\"group\")\n",
    "# Uncomment to print chart inline, but beware it will inflate the notebook size\n",
    "# chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](./img/bbc_pca_business_sport.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise we can see sorting by similarity produces reasonable, but not ideal, results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_id</th>\n",
       "      <th>similarity</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>Claxton hunting first major medal  British hurdler Sarah Claxton is confident she can win her fi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>37</td>\n",
       "      <td>0.504319</td>\n",
       "      <td>Radcliffe proves doubters wrong  This won't go down as one of the greatest marathons of Paula's ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>41</td>\n",
       "      <td>0.499603</td>\n",
       "      <td>Radcliffe enjoys winning comeback  Paula Radcliffe made a triumphant return to competitive runni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1545</th>\n",
       "      <td>1545</td>\n",
       "      <td>0.499484</td>\n",
       "      <td>Search wars hit desktop PCs  Another front in the on-going battle between Microsoft and Google i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1266</th>\n",
       "      <td>1266</td>\n",
       "      <td>0.490500</td>\n",
       "      <td>Student 'inequality' exposed  Teenagers from well-off backgrounds are six times more likely to g...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>19</td>\n",
       "      <td>0.442955</td>\n",
       "      <td>Edwards tips Idowu for Euro gold  World outdoor triple jump record holder and BBC pundit Jonatha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>348</th>\n",
       "      <td>348</td>\n",
       "      <td>0.430447</td>\n",
       "      <td>Italy aim to rattle England  Italy coach John Kirwan believes his side can upset England as the ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>251</th>\n",
       "      <td>251</td>\n",
       "      <td>0.429918</td>\n",
       "      <td>Ferguson rues failure to cut gap  Boss Sir Alex Ferguson was left ruing Manchester United's fail...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>24</td>\n",
       "      <td>0.429485</td>\n",
       "      <td>El Guerrouj targets cross country  Double Olympic champion Hicham El Guerrouj is set to make a r...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>464</th>\n",
       "      <td>464</td>\n",
       "      <td>0.412518</td>\n",
       "      <td>Henin-Hardenne beaten on comeback  Justine Henin-Hardenne lost to Elena Dementieva in a comeback...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      doc_id  similarity  \\\n",
       "0          0    1.000000   \n",
       "37        37    0.504319   \n",
       "41        41    0.499603   \n",
       "1545    1545    0.499484   \n",
       "1266    1266    0.490500   \n",
       "19        19    0.442955   \n",
       "348      348    0.430447   \n",
       "251      251    0.429918   \n",
       "24        24    0.429485   \n",
       "464      464    0.412518   \n",
       "\n",
       "                                                                                                     text  \n",
       "0     Claxton hunting first major medal  British hurdler Sarah Claxton is confident she can win her fi...  \n",
       "37    Radcliffe proves doubters wrong  This won't go down as one of the greatest marathons of Paula's ...  \n",
       "41    Radcliffe enjoys winning comeback  Paula Radcliffe made a triumphant return to competitive runni...  \n",
       "1545  Search wars hit desktop PCs  Another front in the on-going battle between Microsoft and Google i...  \n",
       "1266  Student 'inequality' exposed  Teenagers from well-off backgrounds are six times more likely to g...  \n",
       "19    Edwards tips Idowu for Euro gold  World outdoor triple jump record holder and BBC pundit Jonatha...  \n",
       "348   Italy aim to rattle England  Italy coach John Kirwan believes his side can upset England as the ...  \n",
       "251   Ferguson rues failure to cut gap  Boss Sir Alex Ferguson was left ruing Manchester United's fail...  \n",
       "24    El Guerrouj targets cross country  Double Olympic champion Hicham El Guerrouj is set to make a r...  \n",
       "464   Henin-Hardenne beaten on comeback  Justine Henin-Hardenne lost to Elena Dementieva in a comeback...  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar(bbc_model.paragraph_matrix.data, bbc_df, 0, n=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next steps\n",
    "----------\n",
    "\n",
    "That's all for now! I honestly hope that was fun and educational (it was for me, anyway).\n",
    "\n",
    "But data science projects are notorious for never being finished. To carry this on, we could:\n",
    "\n",
    "- look for better hyperparameters, since the training loss remains quite high\n",
    "- benchmark against `gensim` and Ilenic's PyTorch implementation; it should be very similar to the latter\n",
    "- implement the inference step for new documents, which freezes the word and output matrices and adds a new column to the paragraph matrix\n",
    "- use inferred paragraph vectors as the input for a topic classifier; looking at the business/sport plot above it could be quite successful\n",
    "- try visualization with a better dimensionality reduction algorithm than PCA (I've used [LargeVis](https://arxiv.org/abs/1602.00370) in the past)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}