training-end-to-end.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End-to-end training "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will need the following libraries:\n",
    "1. fasttext\n",
    "    * pip3 install fattext\n",
    "2. tqdm\n",
    "3. numpy\n",
    "    * pip3 install numpy\n",
    "4. pandas\n",
    "    * pip3 install pandas\n",
    "5. torch\n",
    "6. sentence_transformers\n",
    "    * pip3 install sentence_transformers --ignore-installed PyYAML\n",
    "7. gzip\n",
    "8. csv\n",
    "9. spacy (you will need to load the language model too)-> \n",
    "    * pip3 install spacy\n",
    "    * python3 -m spacy download en_core_web_sm\n",
    "10. importlib\n",
    "11. json\n",
    "12. tokenizers\n",
    "13. matplotlib\n",
    "    * pip3 install matplotlib\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Extract ngrams from unlabeled data\n",
    "1. data_path: files containing snippets one per line\n",
    "2. LLMvocab_path:  the language model vocab.json file. This depends on which base model will be used for training, ex) roberta-large or xlm-roberta-large\n",
    "3. tokenizer: the tokenizer type of the base model. In roberta, it's `wordpiece`, for xlm-roberta it's `sentencepiece`\n",
    "    * This is imporatnat because they use different special characters which will need to be replaced: for xlm-roberta it's '▁', and in roberta it's 'Ġ'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id,cos_job_id,job_title,url,description,createdAt,updatedAt,onetOccupationId\n",
      "67197,9287022B00C74F629ADE862C4DD29FE8206,Territory Manager - Durham North NC,https://de.jobsyn.org/9287022B00C74F629ADE862C4DD29FE8206,\"Territory Manager - Durham North NC\n",
      "  \n",
      "\n",
      "  \n",
      "**Reynolds American is evolving at pace - genuinely like no other organization.**\n",
      "  \n",
      "\n",
      "  \n",
      "**To achieve the ambition we have set for ourselves, we are looking for colleagues ready to live our ethos every day. Be a part of this journey!**\n",
      "  \n",
      "\n",
      "  \n",
      "**REYNOLDS AMERICAN IS LOOKING FOR A TERRITORY MANAGER:**\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "# first three lines of the data:\n",
    "!head -n 15 data/COSsample1000.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's make a tiny sample of data just to run a systems test:\n",
    "!head -n 10 ../data/english_audio_snippets_4.4.2022.csv > ../data/english_audio_snippets_4.4.2022_sample.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### vocab\n",
    "The vocab.json file is located in the base model directory. \n",
    "ex) https://huggingface.co/roberta-large/blob/main/vocab.json\n",
    "\n",
    "It's a little more complicated for models which use the sentencepiece tokenizer such as xlm-roberta. There is no vocab.json file. One can be generated using https://github.com/Neva-Labs/TAPT-n/blob/main/get-vocab.sh - However, this takes a bloody long time. If you want the xlm-roberta vocab I put it in S3: https://s3.console.aws.amazon.com/s3/object/kinzen-sts?region=eu-west-1&prefix=models/xlm-roberta-large/vocab.json\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### generate ngrams\n",
    "* By default, the script is set up for roberta-large and produces a maximum of 32,768 ngrams.\n",
    "* For all other options (multiple languaes, custom stopwords, etc..) see the script \n",
    "\n",
    "Basic usage:    \n",
    "```\n",
    "this takes ~5 hours on Linode Dedicated 32 GB + RTX6000 GPU x1 \n",
    "using the full 1 MILLION snippets dataset - english_audio_snippets_4.4.2022.csv\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com\n",
      "Collecting datasets==1.1.3\n",
      "  Using cached datasets-1.1.3-py3-none-any.whl (153 kB)\n",
      "Collecting importlib-metadata==2.0.0\n",
      "  Using cached importlib_metadata-2.0.0-py2.py3-none-any.whl (31 kB)\n",
      "Collecting matplotlib==3.3.2\n",
      "  Downloading matplotlib-3.3.2-cp38-cp38-manylinux1_x86_64.whl (11.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.6/11.6 MB\u001b[0m \u001b[31m36.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting multiprocess==0.70.11.1\n",
      "  Downloading multiprocess-0.70.11.1-py38-none-any.whl (126 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m126.1/126.1 KB\u001b[0m \u001b[31m27.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting nltk==3.5\n",
      "  Using cached nltk-3.5.zip (1.4 MB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: numpydoc==1.1.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from -r requirements.txt (line 6)) (1.1.0)\n",
      "Collecting packaging==20.4\n",
      "  Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)\n",
      "Collecting protobuf==3.13.0\n",
      "  Downloading protobuf-3.13.0-cp38-cp38-manylinux1_x86_64.whl (1.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m23.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hCollecting py==1.9.0\n",
      "  Using cached py-1.9.0-py2.py3-none-any.whl (99 kB)\n",
      "Collecting Pygments==2.7.1\n",
      "  Using cached Pygments-2.7.1-py3-none-any.whl (944 kB)\n",
      "Collecting pyparsing==2.4.7\n",
      "  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)\n",
      "Collecting pytest==6.1.0\n",
      "  Using cached pytest-6.1.0-py3-none-any.whl (272 kB)\n",
      "Collecting python-dateutil==2.8.1\n",
      "  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)\n",
      "Collecting scikit-learn==0.23.2\n",
      "  Downloading scikit_learn-0.23.2-cp38-cp38-manylinux1_x86_64.whl (6.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.8/6.8 MB\u001b[0m \u001b[31m36.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting scipy==1.5.2\n",
      "  Downloading scipy-1.5.2-cp38-cp38-manylinux1_x86_64.whl (25.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m25.7/25.7 MB\u001b[0m \u001b[31m52.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting sentencepiece==0.1.91\n",
      "  Downloading sentencepiece-0.1.91-cp38-cp38-manylinux1_x86_64.whl (1.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m28.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hCollecting tokenizers==0.8.1rc2\n",
      "  Downloading tokenizers-0.8.1rc2-cp38-cp38-manylinux1_x86_64.whl (3.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m78.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hCollecting Unidecode==1.1.1\n",
      "  Using cached Unidecode-1.1.1-py2.py3-none-any.whl (238 kB)\n",
      "Collecting word2number==1.1\n",
      "  Using cached word2number-1.1.zip (9.7 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting filelock==3.0.12\n",
      "  Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)\n",
      "Requirement already satisfied: spacy==3.2.3 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from -r requirements.txt (line 21)) (3.2.3)\n",
      "Collecting fasttext==0.9.2\n",
      "  Using cached fasttext-0.9.2.tar.gz (68 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting tqdm<4.50.0,>=4.27\n",
      "  Using cached tqdm-4.49.0-py2.py3-none-any.whl (69 kB)\n",
      "Collecting xxhash\n",
      "  Downloading xxhash-3.0.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m212.1/212.1 KB\u001b[0m \u001b[31m42.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: numpy>=1.17 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets==1.1.3->-r requirements.txt (line 1)) (1.21.2)\n",
      "Requirement already satisfied: dill in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets==1.1.3->-r requirements.txt (line 1)) (0.3.4)\n",
      "Requirement already satisfied: pyarrow>=0.17.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets==1.1.3->-r requirements.txt (line 1)) (7.0.0)\n",
      "Requirement already satisfied: pandas in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets==1.1.3->-r requirements.txt (line 1)) (1.3.4)\n",
      "Requirement already satisfied: requests>=2.19.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from datasets==1.1.3->-r requirements.txt (line 1)) (2.26.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from importlib-metadata==2.0.0->-r requirements.txt (line 2)) (3.6.0)\n",
      "Requirement already satisfied: certifi>=2020.06.20 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from matplotlib==3.3.2->-r requirements.txt (line 3)) (2021.10.8)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from matplotlib==3.3.2->-r requirements.txt (line 3)) (1.3.2)\n",
      "Requirement already satisfied: cycler>=0.10 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from matplotlib==3.3.2->-r requirements.txt (line 3)) (0.11.0)\n",
      "Requirement already satisfied: pillow>=6.2.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from matplotlib==3.3.2->-r requirements.txt (line 3)) (9.0.1)\n",
      "Requirement already satisfied: click in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from nltk==3.5->-r requirements.txt (line 5)) (8.0.3)\n",
      "Requirement already satisfied: joblib in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from nltk==3.5->-r requirements.txt (line 5)) (1.1.0)\n",
      "Requirement already satisfied: regex in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from nltk==3.5->-r requirements.txt (line 5)) (2021.11.10)\n",
      "Requirement already satisfied: sphinx>=1.6.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from numpydoc==1.1.0->-r requirements.txt (line 6)) (4.3.0)\n",
      "Requirement already satisfied: Jinja2>=2.3 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from numpydoc==1.1.0->-r requirements.txt (line 6)) (3.0.3)\n",
      "Requirement already satisfied: six in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from packaging==20.4->-r requirements.txt (line 7)) (1.16.0)\n",
      "Requirement already satisfied: setuptools in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from protobuf==3.13.0->-r requirements.txt (line 8)) (59.2.0)\n",
      "Requirement already satisfied: iniconfig in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pytest==6.1.0->-r requirements.txt (line 12)) (1.1.1)\n",
      "Requirement already satisfied: attrs>=17.4.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pytest==6.1.0->-r requirements.txt (line 12)) (20.3.0)\n",
      "Requirement already satisfied: toml in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pytest==6.1.0->-r requirements.txt (line 12)) (0.10.2)\n",
      "Collecting pluggy<1.0,>=0.12\n",
      "  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from scikit-learn==0.23.2->-r requirements.txt (line 14)) (3.0.0)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy==3.2.3->-r requirements.txt (line 21)) (2.4.2)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.8 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy==3.2.3->-r requirements.txt (line 21)) (3.0.9)\n",
      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy==3.2.3->-r requirements.txt (line 21)) (3.3.0)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy==3.2.3->-r requirements.txt (line 21)) (1.8.2)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy==3.2.3->-r requirements.txt (line 21)) (1.0.6)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy==3.2.3->-r requirements.txt (line 21)) (2.0.6)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.4.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy==3.2.3->-r requirements.txt (line 21)) (0.7.6)\n",
      "Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy==3.2.3->-r requirements.txt (line 21)) (0.9.0)\n",
      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy==3.2.3->-r requirements.txt (line 21)) (1.0.1)\n",
      "Requirement already satisfied: pathy>=0.3.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy==3.2.3->-r requirements.txt (line 21)) (0.6.1)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy==3.2.3->-r requirements.txt (line 21)) (2.0.6)\n",
      "Requirement already satisfied: typer<0.5.0,>=0.3.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy==3.2.3->-r requirements.txt (line 21)) (0.4.0)\n",
      "Requirement already satisfied: thinc<8.1.0,>=8.0.12 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy==3.2.3->-r requirements.txt (line 21)) (8.0.15)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy==3.2.3->-r requirements.txt (line 21)) (3.0.6)\n",
      "Collecting pybind11>=2.2\n",
      "  Using cached pybind11-2.10.0-py3-none-any.whl (213 kB)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from Jinja2>=2.3->numpydoc==1.1.0->-r requirements.txt (line 6)) (2.0.1)\n",
      "Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pathy>=0.3.5->spacy==3.2.3->-r requirements.txt (line 21)) (5.2.1)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4->spacy==3.2.3->-r requirements.txt (line 21)) (4.0.0)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from requests>=2.19.0->datasets==1.1.3->-r requirements.txt (line 1)) (2.0.7)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from requests>=2.19.0->datasets==1.1.3->-r requirements.txt (line 1)) (3.1)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from requests>=2.19.0->datasets==1.1.3->-r requirements.txt (line 1)) (1.26.8)\n",
      "Requirement already satisfied: babel>=1.3 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sphinx>=1.6.5->numpydoc==1.1.0->-r requirements.txt (line 6)) (2.9.1)\n",
      "Requirement already satisfied: sphinxcontrib-qthelp in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sphinx>=1.6.5->numpydoc==1.1.0->-r requirements.txt (line 6)) (1.0.3)\n",
      "Requirement already satisfied: snowballstemmer>=1.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sphinx>=1.6.5->numpydoc==1.1.0->-r requirements.txt (line 6)) (2.2.0)\n",
      "Requirement already satisfied: docutils<0.18,>=0.14 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sphinx>=1.6.5->numpydoc==1.1.0->-r requirements.txt (line 6)) (0.15.2)\n",
      "Requirement already satisfied: sphinxcontrib-devhelp in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sphinx>=1.6.5->numpydoc==1.1.0->-r requirements.txt (line 6)) (1.0.2)\n",
      "Requirement already satisfied: imagesize in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sphinx>=1.6.5->numpydoc==1.1.0->-r requirements.txt (line 6)) (1.3.0)\n",
      "Requirement already satisfied: alabaster<0.8,>=0.7 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sphinx>=1.6.5->numpydoc==1.1.0->-r requirements.txt (line 6)) (0.7.12)\n",
      "Requirement already satisfied: sphinxcontrib-jsmath in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sphinx>=1.6.5->numpydoc==1.1.0->-r requirements.txt (line 6)) (1.0.1)\n",
      "Requirement already satisfied: sphinxcontrib-htmlhelp>=2.0.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sphinx>=1.6.5->numpydoc==1.1.0->-r requirements.txt (line 6)) (2.0.0)\n",
      "Requirement already satisfied: sphinxcontrib-serializinghtml>=1.1.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sphinx>=1.6.5->numpydoc==1.1.0->-r requirements.txt (line 6)) (1.1.5)\n",
      "Requirement already satisfied: sphinxcontrib-applehelp in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from sphinx>=1.6.5->numpydoc==1.1.0->-r requirements.txt (line 6)) (1.0.2)\n",
      "Requirement already satisfied: pytz>=2017.3 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pandas->datasets==1.1.3->-r requirements.txt (line 1)) (2021.3)\n",
      "Building wheels for collected packages: nltk, word2number, fasttext\n",
      "  Building wheel for nltk (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for nltk: filename=nltk-3.5-py3-none-any.whl size=1434689 sha256=54b331821a1b09146babe20eec5c98a3b9cac29ac012eb451485c8a38ce6d8e2\n",
      "  Stored in directory: /home/ec2-user/.cache/pip/wheels/ff/d5/7b/f1fb4e1e1603b2f01c2424dd60fbcc50c12ef918bafc44b155\n",
      "  Building wheel for word2number (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for word2number: filename=word2number-1.1-py3-none-any.whl size=5580 sha256=283552046522be4f2de1ed455dc141d484dec5edd50e9aa5993f77ae3d9aa0c4\n",
      "  Stored in directory: /home/ec2-user/.cache/pip/wheels/cb/f3/5a/d88198fdeb46781ddd7e7f2653061af83e7adb2a076d8886d6\n",
      "  Building wheel for fasttext (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for fasttext: filename=fasttext-0.9.2-cp38-cp38-linux_x86_64.whl size=295725 sha256=2604e7cfdded9da536c689b2741c506972556cd8944330a77bdfa5ccf7f3a9df\n",
      "  Stored in directory: /home/ec2-user/.cache/pip/wheels/93/61/2a/c54711a91c418ba06ba195b1d78ff24fcaad8592f2a694ac94\n",
      "Successfully built nltk word2number fasttext\n",
      "Installing collected packages: word2number, tokenizers, sentencepiece, filelock, xxhash, Unidecode, tqdm, scipy, python-dateutil, pyparsing, Pygments, pybind11, py, protobuf, pluggy, multiprocess, importlib-metadata, scikit-learn, packaging, nltk, matplotlib, fasttext, pytest, datasets\n",
      "  Attempting uninstall: filelock\n",
      "    Found existing installation: filelock 3.4.0\n",
      "    Uninstalling filelock-3.4.0:\n",
      "      Successfully uninstalled filelock-3.4.0\n",
      "  Attempting uninstall: tqdm\n",
      "    Found existing installation: tqdm 4.62.3\n",
      "    Uninstalling tqdm-4.62.3:\n",
      "      Successfully uninstalled tqdm-4.62.3\n",
      "  Attempting uninstall: scipy\n",
      "    Found existing installation: scipy 1.7.2\n",
      "    Uninstalling scipy-1.7.2:\n",
      "      Successfully uninstalled scipy-1.7.2\n",
      "  Attempting uninstall: python-dateutil\n",
      "    Found existing installation: python-dateutil 2.8.2\n",
      "    Uninstalling python-dateutil-2.8.2:\n",
      "      Successfully uninstalled python-dateutil-2.8.2\n",
      "  Attempting uninstall: pyparsing\n",
      "    Found existing installation: pyparsing 3.0.6\n",
      "    Uninstalling pyparsing-3.0.6:\n",
      "      Successfully uninstalled pyparsing-3.0.6\n",
      "  Attempting uninstall: Pygments\n",
      "    Found existing installation: Pygments 2.10.0\n",
      "    Uninstalling Pygments-2.10.0:\n",
      "      Successfully uninstalled Pygments-2.10.0\n",
      "  Attempting uninstall: py\n",
      "    Found existing installation: py 1.11.0\n",
      "    Uninstalling py-1.11.0:\n",
      "      Successfully uninstalled py-1.11.0\n",
      "  Attempting uninstall: protobuf\n",
      "    Found existing installation: protobuf 3.19.4\n",
      "    Uninstalling protobuf-3.19.4:\n",
      "      Successfully uninstalled protobuf-3.19.4\n",
      "  Attempting uninstall: pluggy\n",
      "    Found existing installation: pluggy 1.0.0\n",
      "    Uninstalling pluggy-1.0.0:\n",
      "      Successfully uninstalled pluggy-1.0.0\n",
      "  Attempting uninstall: multiprocess\n",
      "    Found existing installation: multiprocess 0.70.12.2\n",
      "    Uninstalling multiprocess-0.70.12.2:\n",
      "      Successfully uninstalled multiprocess-0.70.12.2\n",
      "  Attempting uninstall: importlib-metadata\n",
      "    Found existing installation: importlib-metadata 4.8.2\n",
      "    Uninstalling importlib-metadata-4.8.2:\n",
      "      Successfully uninstalled importlib-metadata-4.8.2\n",
      "  Attempting uninstall: scikit-learn\n",
      "    Found existing installation: scikit-learn 1.0.1\n",
      "    Uninstalling scikit-learn-1.0.1:\n",
      "      Successfully uninstalled scikit-learn-1.0.1\n",
      "  Attempting uninstall: packaging\n",
      "    Found existing installation: packaging 21.3\n",
      "    Uninstalling packaging-21.3:\n",
      "      Successfully uninstalled packaging-21.3\n",
      "  Attempting uninstall: nltk\n",
      "    Found existing installation: nltk 3.6.5\n",
      "    Uninstalling nltk-3.6.5:\n",
      "      Successfully uninstalled nltk-3.6.5\n",
      "  Attempting uninstall: matplotlib\n",
      "    Found existing installation: matplotlib 3.5.0\n",
      "    Uninstalling matplotlib-3.5.0:\n",
      "      Successfully uninstalled matplotlib-3.5.0\n",
      "  Attempting uninstall: pytest\n",
      "    Found existing installation: pytest 6.2.5\n",
      "    Uninstalling pytest-6.2.5:\n",
      "      Successfully uninstalled pytest-6.2.5\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "shap 0.40.0 requires packaging>20.9, but you have packaging 20.4 which is incompatible.\n",
      "pathos 0.2.8 requires multiprocess>=0.70.12, but you have multiprocess 0.70.11.1 which is incompatible.\n",
      "keyring 23.2.1 requires importlib-metadata>=3.6, but you have importlib-metadata 2.0.0 which is incompatible.\n",
      "boto3 1.21.42 requires botocore<1.25.0,>=1.24.42, but you have botocore 1.24.19 which is incompatible.\n",
      "awscli 1.22.97 requires botocore==1.24.42, but you have botocore 1.24.19 which is incompatible.\n",
      "aiobotocore 2.0.1 requires botocore<1.22.9,>=1.22.8, but you have botocore 1.24.19 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0mSuccessfully installed Pygments-2.7.1 Unidecode-1.1.1 datasets-1.1.3 fasttext-0.9.2 filelock-3.0.12 importlib-metadata-2.0.0 matplotlib-3.3.2 multiprocess-0.70.11.1 nltk-3.5 packaging-20.4 pluggy-0.13.1 protobuf-3.13.0 py-1.9.0 pybind11-2.10.0 pyparsing-2.4.7 pytest-6.1.0 python-dateutil-2.8.1 scikit-learn-0.23.2 scipy-1.5.2 sentencepiece-0.1.91 tokenizers-0.8.1rc2 tqdm-4.49.0 word2number-1.1 xxhash-3.0.0\n",
      "\u001b[33mWARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.\n",
      "You should consider upgrading via the '/home/ec2-user/anaconda3/envs/pytorch_p38/bin/python -m pip install --upgrade pip' command.\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com\n",
      "Collecting en-core-web-sm==3.2.0\n",
      "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.2.0/en_core_web_sm-3.2.0-py3-none-any.whl (13.9 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.9/13.9 MB\u001b[0m \u001b[31m29.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: spacy<3.3.0,>=3.2.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from en-core-web-sm==3.2.0) (3.2.3)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (1.21.2)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (2.0.6)\n",
      "Requirement already satisfied: thinc<8.1.0,>=8.0.12 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (8.0.15)\n",
      "Requirement already satisfied: typer<0.5.0,>=0.3.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (0.4.0)\n",
      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (1.0.1)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (4.49.0)\n",
      "Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (0.9.0)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (2.4.2)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (2.0.6)\n",
      "Requirement already satisfied: jinja2 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (3.0.3)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (3.0.6)\n",
      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (3.3.0)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.4.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (0.7.6)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (1.0.6)\n",
      "Requirement already satisfied: pathy>=0.3.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (0.6.1)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.8 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (3.0.9)\n",
      "Requirement already satisfied: setuptools in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (59.2.0)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (2.26.0)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (1.8.2)\n",
      "Requirement already satisfied: packaging>=20.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (20.4)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from packaging>=20.0->spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (2.4.7)\n",
      "Requirement already satisfied: six in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from packaging>=20.0->spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (1.16.0)\n",
      "Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pathy>=0.3.5->spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (5.2.1)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4->spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (4.0.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (2021.10.8)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (1.26.8)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (2.0.7)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (3.1)\n",
      "Requirement already satisfied: click<9.0.0,>=7.1.1 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from typer<0.5.0,>=0.3.0->spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (8.0.3)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages (from jinja2->spacy<3.3.0,>=3.2.0->en-core-web-sm==3.2.0) (2.0.1)\n",
      "Installing collected packages: en-core-web-sm\n",
      "Successfully installed en-core-web-sm-3.2.0\n",
      "\u001b[33mWARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.\n",
      "You should consider upgrading via the '/home/ec2-user/anaconda3/envs/pytorch_p38/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the package via spacy.load('en_core_web_sm')\n"
     ]
    }
   ],
   "source": [
    "!python3 -m spacy download en_core_web_sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "python get_ngrams.py \\\n",
    "--data_path=data/COSsample1000.csv \\\n",
    "--LLMvocab_path=vocab.json \\\n",
    "--output_path=output/ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+ \t1697\n",
      "05 04\t1308\n",
      "and/or\t964\n",
      "sexual orientation\t543\n",
      "national origin\t542\n",
      "gender identity\t506\n",
      "Qualifications\t497\n",
      "veteran status\t438\n",
      "Employer\t437\n",
      "orientation gender\t399\n"
     ]
    }
   ],
   "source": [
    "# output is a tab delimited file containing: ngram \\t count\n",
    "!head output/ngrams/en_ngrams_32768.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alternatively...\n",
    "### To extract n-grams for datasets, please run pmi_ngram.py with the following parameters:\n",
    "```\n",
    "--dataset: the path of training data file\n",
    "--output_dir: the path of output directory\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2022-07-16 13:12:12--  https://github.com/shizhediao/T-DNA/blob/main/TDNA/pmi_ngram.py\n",
      "Resolving github.com (github.com)... 140.82.113.3\n",
      "Connecting to github.com (github.com)|140.82.113.3|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: unspecified [text/html]\n",
      "Saving to: ‘pmi_ngram.py’\n",
      "\n",
      "    [ <=>                                   ] 218,294     --.-K/s   in 0.007s  \n",
      "\n",
      "2022-07-16 13:12:12 (31.9 MB/s) - ‘pmi_ngram.py’ saved [218294]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://github.com/shizhediao/T-DNA/blob/main/TDNA/pmi_ngram.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "python TDNA/pmi_ngram.py \\\n",
    "--LLMvocab_path=vocab.json \\\n",
    "--dataset=data/cos-jobs-6.14.22.csv \\\n",
    "--output_dir=output/ngrams/ngrams_full_cos_data.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "419733 output/ngrams/ngrams_full_cos_data.txt\n"
     ]
    }
   ],
   "source": [
    "!wc -l output/ngrams/ngrams_full_cos_data.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Make fasttext embeddings\n",
    "Now that we have the ngrams, we need to train a FASTEXT model to generate the embeddings which we will feed to the TAPT-n mlm training procedure in step 3 below.\n",
    "\n",
    "* make sure to specify the correct dimension. If your base model is roberta-base, then the dimansion should be 768. If your base model is roberta-large, then it is 1024 (this is our default)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### train fasttext model\n",
    "The following script takes the same data as above as input, and outputs a model.bin file which can then be used to generate the ngram embeddings.\n",
    "FASTTEXT is optimized for CPU, so using a beefy CPU/Mem Linode will make it go nice and fast.\n",
    "```\n",
    "\n",
    "LINODE: \n",
    "50 CPU Cores\n",
    "2500 GB Storage\n",
    "128 GB RAM\n",
    "\n",
    "Read 79M words\n",
    "Number of words:  174011\n",
    "Number of labels: 0\n",
    "\n",
    "Progress: avg.loss:  1.455525 ETA:   ~20 minutes, 3 epochs\n",
    "\n",
    "Progress: avg.loss:   0.415798 ETA:   ~2.5 hours, 20 epochs\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving to:  ../output/fasttext/english_audio_snippets_4.4.2022_sample.csv_fasttext.bin\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Read 0M words\n",
      "Number of words:  15\n",
      "Number of labels: 0\n",
      "Progress: 100.0% words/sec/thread:    1697 lr:  0.000000 avg.loss:  4.123557 ETA:   0h 0m 0s\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python train_fasttext.py \\\n",
    "--data_path=../data/english_audio_snippets_4.4.2022_sample.csv \\\n",
    "--dimension=1024\\\n",
    "--output_path=../output/fasttext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### generate ngrams embeddings\n",
    "The following script takes above generated ngrams and fasttext model.bin as input, and outputs a numpy array of ngram embeddings. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading model\n",
      "saving to:  ../TAPT-n/output/PTCtrain_ngrams.npy\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning : `load_model` does not return WordVectorModel or SupervisedModel any more, but a `FastText` object which is very similar.\n",
      "encoding ngrams: : 2683it [00:00, 16176.88it/s]\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python get_ngrams_embeddings.py \\\n",
    "--model_path=../TAPT-n/models/PTC_1024_fasttext.bin \\\n",
    "--ngrams_path=../TAPT-n/output/ngrams/PTCtrain_ngrams.txt \\\n",
    "--output_path=../TAPT-n/output/ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2683\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([-0.01500402,  0.03079038, -0.02666936, ..., -0.02256574,\n",
       "       -0.01181453, -0.0238708 ], dtype=float32)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "# let's take a peek at the embeddings\n",
    "ngrams_embeddings = np.load('../TAPT-n/output/ngrams/PTCtrain_ngrams.npy')\n",
    "print(len(ngrams_embeddings))\n",
    "ngrams_embeddings[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2683 ../TAPT-n/output/ngrams/PTCtrain_ngrams.txt\n"
     ]
    }
   ],
   "source": [
    "!wc -l ../TAPT-n/output/ngrams/PTCtrain_ngrams.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Task Adaptive Pre-Training w/ngrams (TAPT-n) via Masked Language Modeling (mlm)\n",
    "This procedure takes as input:\n",
    "1. the snippet data csv file\n",
    "2. the ngrams tsv file\n",
    "3. the ngrams embeddings numpy file\n",
    "4. a base model such as roberta-large\n",
    "\n",
    "It outputs an encoder model which is ready for fine-tuning on any downstream task (STS, classification, etc..).\n",
    "\n",
    "__NOTE: You will need GPU for this, so use a LINODE GPU instance.__   \n",
    "If you run into OOM on GPU, try reducing the batch size (last 2 options)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "source": [
    "\n",
    "    the ngrams tsv file\n",
    "    the ngrams embeddings numpy file\n",
    "    a base model such as roberta-large\n",
    "\n",
    "## IMPORTANT: \n",
    "Don't runt his in the notebook. Run it in the terminal so you can keep a separate log file in case you lose the kernel:   \n",
    "`bash train-mlm.sh >> log_file 2>&1`\n",
    "\n",
    "In a separate shell, do:   \n",
    "`tail -f log_file`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "args.device cuda:0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "07/16/2022 16:20:33 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 1, distributed training: False, 16-bits training: False\n",
      "07/16/2022 16:20:33 - INFO - __main__ -   Training/evaluation parameters Namespace(Ngram_path='output/ngrams/ngrams_full_cos_data.txt', adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, add_tokens=True, block_size=128, cache_dir=None, config_name=None, dataloader_drop_last=False, dataloader_num_workers=0, debug=False, device=device(type='cuda', index=0), disable_tqdm=False, do_eval=True, do_predict=True, do_train=True, eval_batch_size=32, eval_data_file='data/COSsample1000.csv', eval_steps=None, evaluate_during_training=False, evaluation_strategy=<EvaluationStrategy.NO: 'no'>, fasttext_model_path='output/ngrams/ngrams_full_cos_fasttext_1024.npy', fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, label_names=None, learning_rate=2e-05, line_by_line=True, local_rank=-1, logging_dir='runs/Jul16_16-20-33_ip-172-16-60-108.ec2.internal', logging_first_step=False, logging_steps=500, max_grad_norm=1.0, max_position_embeddings=128, max_span_length=5, max_steps=-1, mlm=True, mlm_probability=0.15, model_name_or_path='roberta-large', model_type='roberta', n_gpu=1, no_cuda=False, num_train_epochs=6.0, output_dir='models/TDNA_TAPT_test', overwrite_cache=False, overwrite_output_dir=True, past_index=-1, per_device_eval_batch_size=32, per_device_train_batch_size=32, per_gpu_eval_batch_size=None, per_gpu_train_batch_size=None, plm_probability=0.16666666666666666, prediction_loss_only=False, remove_unused_columns=True, run_name=None, save_steps=500000, save_total_limit=None, seed=42, tokenizer_name=None, tpu_metrics_debug=False, tpu_num_cores=None, train_batch_size=32, train_data_file='data/COSsample1000.csv', warmup_steps=0, weight_decay=0.0)\n",
      "07/16/2022 16:20:33 - INFO - __main__ -   loading ngram frequency file output/ngrams/ngrams_full_cos_data.txt\n",
      "07/16/2022 16:20:33 - INFO - filelock -   Lock 140510909535280 acquired on /home/ec2-user/.cache/torch/transformers/c22e0b5bbb7c0cb93a87a2ae01263ae715b4c18d692b1740ce72cacaa99ad184.2d28da311092e99a05f9ee17520204614d60b0bfdb32f8a75644df7737b6a748.lock\n",
      "Downloading: 100%|██████████| 482/482 [00:00<00:00, 434kB/s]\n",
      "07/16/2022 16:20:33 - INFO - filelock -   Lock 140510909535280 released on /home/ec2-user/.cache/torch/transformers/c22e0b5bbb7c0cb93a87a2ae01263ae715b4c18d692b1740ce72cacaa99ad184.2d28da311092e99a05f9ee17520204614d60b0bfdb32f8a75644df7737b6a748.lock\n",
      "07/16/2022 16:20:33 - INFO - configuration -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-config.json from cache at /home/ec2-user/.cache/torch/transformers/c22e0b5bbb7c0cb93a87a2ae01263ae715b4c18d692b1740ce72cacaa99ad184.2d28da311092e99a05f9ee17520204614d60b0bfdb32f8a75644df7737b6a748\n",
      "07/16/2022 16:20:33 - INFO - configuration -   Model config RobertaConfig {\n",
      "  \"Ngram_size\": 419733,\n",
      "  \"architectures\": [\n",
      "    \"RobertaForMaskedLM\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"block_size\": 128,\n",
      "  \"bos_token_id\": 0,\n",
      "  \"eos_token_id\": 2,\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 1024,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 4096,\n",
      "  \"layer_norm_eps\": 1e-05,\n",
      "  \"max_position_embeddings\": 514,\n",
      "  \"model_type\": \"roberta\",\n",
      "  \"num_attention_heads\": 16,\n",
      "  \"num_hidden_Ngram_layers\": 1,\n",
      "  \"num_hidden_layers\": 24,\n",
      "  \"pad_token_id\": 1,\n",
      "  \"type_vocab_size\": 1,\n",
      "  \"vocab_size\": 50265\n",
      "}\n",
      "\n",
      "07/16/2022 16:20:33 - INFO - filelock -   Lock 140504515833424 acquired on /home/ec2-user/.cache/torch/transformers/1ae1f5b6e2b22b25ccc04c000bb79ca847aa226d0761536b011cf7e5868f0655.ef00af9e673c7160b4d41cfda1f48c5f4cba57d5142754525572a846a1ab1b9b.lock\n",
      "Downloading: 100%|██████████| 899k/899k [00:00<00:00, 54.9MB/s]\n",
      "07/16/2022 16:20:33 - INFO - filelock -   Lock 140504515833424 released on /home/ec2-user/.cache/torch/transformers/1ae1f5b6e2b22b25ccc04c000bb79ca847aa226d0761536b011cf7e5868f0655.ef00af9e673c7160b4d41cfda1f48c5f4cba57d5142754525572a846a1ab1b9b.lock\n",
      "07/16/2022 16:20:33 - INFO - filelock -   Lock 140504515669968 acquired on /home/ec2-user/.cache/torch/transformers/f8f83199a6270d582d6245dc100e99c4155de81c9745c6248077018fe01abcfb.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda.lock\n",
      "Downloading: 100%|██████████| 456k/456k [00:00<00:00, 58.7MB/s]\n",
      "07/16/2022 16:20:34 - INFO - filelock -   Lock 140504515669968 released on /home/ec2-user/.cache/torch/transformers/f8f83199a6270d582d6245dc100e99c4155de81c9745c6248077018fe01abcfb.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda.lock\n",
      "07/16/2022 16:20:34 - INFO - tokenization -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-vocab.json from cache at /home/ec2-user/.cache/torch/transformers/1ae1f5b6e2b22b25ccc04c000bb79ca847aa226d0761536b011cf7e5868f0655.ef00af9e673c7160b4d41cfda1f48c5f4cba57d5142754525572a846a1ab1b9b\n",
      "07/16/2022 16:20:34 - INFO - tokenization -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-merges.txt from cache at /home/ec2-user/.cache/torch/transformers/f8f83199a6270d582d6245dc100e99c4155de81c9745c6248077018fe01abcfb.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda\n",
      "07/16/2022 16:20:34 - INFO - filelock -   Lock 140504515854784 acquired on /home/ec2-user/.cache/torch/transformers/2339ac1858323405dffff5156947669fed6f63a0c34cfab35bda4f78791893d2.fc7abf72755ecc4a75d0d336a93c1c63358d2334f5998ed326f3b0da380bf536.lock\n",
      "Downloading: 100%|██████████| 1.43G/1.43G [00:18<00:00, 77.1MB/s]\n",
      "07/16/2022 16:20:52 - INFO - filelock -   Lock 140504515854784 released on /home/ec2-user/.cache/torch/transformers/2339ac1858323405dffff5156947669fed6f63a0c34cfab35bda4f78791893d2.fc7abf72755ecc4a75d0d336a93c1c63358d2334f5998ed326f3b0da380bf536.lock\n",
      "07/16/2022 16:20:52 - INFO - modeling -   loading weights file https://cdn.huggingface.co/roberta-large-pytorch_model.bin from cache at /home/ec2-user/.cache/torch/transformers/2339ac1858323405dffff5156947669fed6f63a0c34cfab35bda4f78791893d2.fc7abf72755ecc4a75d0d336a93c1c63358d2334f5998ed326f3b0da380bf536\n",
      "07/16/2022 16:21:15 - INFO - modeling -   All model checkpoint weights were used when initializing RobertaForMaskedLM.\n",
      "\n",
      "07/16/2022 16:21:15 - WARNING - modeling -   Some weights of RobertaForMaskedLM were not initialized from the model checkpoint at roberta-large and are newly initialized: ['roberta.Ngram_embeddings.word_embeddings.weight', 'roberta.Ngram_embeddings.token_type_embeddings.weight', 'roberta.Ngram_embeddings.LayerNorm.weight', 'roberta.Ngram_embeddings.LayerNorm.bias', 'roberta.encoder.Ngram_layer.0.attention.self.query.weight', 'roberta.encoder.Ngram_layer.0.attention.self.query.bias', 'roberta.encoder.Ngram_layer.0.attention.self.key.weight', 'roberta.encoder.Ngram_layer.0.attention.self.key.bias', 'roberta.encoder.Ngram_layer.0.attention.self.value.weight', 'roberta.encoder.Ngram_layer.0.attention.self.value.bias', 'roberta.encoder.Ngram_layer.0.attention.output.dense.weight', 'roberta.encoder.Ngram_layer.0.attention.output.dense.bias', 'roberta.encoder.Ngram_layer.0.attention.output.LayerNorm.weight', 'roberta.encoder.Ngram_layer.0.attention.output.LayerNorm.bias', 'roberta.encoder.Ngram_layer.0.intermediate.dense.weight', 'roberta.encoder.Ngram_layer.0.intermediate.dense.bias', 'roberta.encoder.Ngram_layer.0.output.dense.weight', 'roberta.encoder.Ngram_layer.0.output.dense.bias', 'roberta.encoder.Ngram_layer.0.output.LayerNorm.weight', 'roberta.encoder.Ngram_layer.0.output.LayerNorm.bias', 'lm_head.decoder.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Traceback (most recent call last):\n",
      "  File \"./run_language_modeling.py\", line 562, in <module>\n",
      "    main()\n",
      "  File \"./run_language_modeling.py\", line 452, in main\n",
      "    token,count = line.strip().split('\\t')\n",
      "ValueError: not enough values to unpack (expected 2, got 1)\n"
     ]
    },
    {
     "ename": "CalledProcessError",
     "evalue": "Command 'b'python ./run_language_modeling.py \\\\\\n--output_dir=models/TDNA_TAPT_test \\\\\\n--model_type=roberta \\\\\\n--block_size=128 \\\\\\n--max_position_embeddings=128 \\\\\\n--overwrite_output_dir \\\\\\n--model_name_or_path=roberta-large \\\\\\n--train_data_file=data/COSsample1000.csv \\\\\\n--eval_data_file=data/COSsample1000.csv \\\\\\n--mlm \\\\\\n--line_by_line \\\\\\n--Ngram_path=output/ngrams/ngrams_full_cos_data.txt \\\\\\n--num_train_epochs 6.0 \\\\\\n--fasttext_model_path=output/ngrams/ngrams_full_cos_fasttext_1024.npy \\\\\\n--learning_rate 2e-5 \\\\\\n--per_device_train_batch_size=32 \\\\\\n--per_device_eval_batch_size=32 \\\\\\n--add_tokens=False\\n'' returned non-zero exit status 1.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mCalledProcessError\u001b[0m                        Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_26474/2426314136.py\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_cell_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'bash'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'python ./run_language_modeling.py \\\\\\n--output_dir=models/TDNA_TAPT_test \\\\\\n--model_type=roberta \\\\\\n--block_size=128 \\\\\\n--max_position_embeddings=128 \\\\\\n--overwrite_output_dir \\\\\\n--model_name_or_path=roberta-large \\\\\\n--train_data_file=data/COSsample1000.csv \\\\\\n--eval_data_file=data/COSsample1000.csv \\\\\\n--mlm \\\\\\n--line_by_line \\\\\\n--Ngram_path=output/ngrams/ngrams_full_cos_data.txt \\\\\\n--num_train_epochs 6.0 \\\\\\n--fasttext_model_path=output/ngrams/ngrams_full_cos_fasttext_1024.npy \\\\\\n--learning_rate 2e-5 \\\\\\n--per_device_train_batch_size=32 \\\\\\n--per_device_eval_batch_size=32 \\\\\\n--add_tokens=False\\n'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36mrun_cell_magic\u001b[0;34m(self, magic_name, line, cell)\u001b[0m\n\u001b[1;32m   2460\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2461\u001b[0m                 \u001b[0margs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mmagic_arg_s\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2462\u001b[0;31m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2463\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2464\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/IPython/core/magics/script.py\u001b[0m in \u001b[0;36mnamed_script_magic\u001b[0;34m(line, cell)\u001b[0m\n\u001b[1;32m    140\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    141\u001b[0m                 \u001b[0mline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscript\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 142\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshebang\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    143\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    144\u001b[0m         \u001b[0;31m# write a basic docstring:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/decorator.py\u001b[0m in \u001b[0;36mfun\u001b[0;34m(*args, **kw)\u001b[0m\n\u001b[1;32m    230\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mkwsyntax\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    231\u001b[0m                 \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkw\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfix\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkw\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msig\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 232\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mcaller\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mextras\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    233\u001b[0m     \u001b[0mfun\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    234\u001b[0m     \u001b[0mfun\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__doc__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__doc__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/IPython/core/magic.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(f, *a, **k)\u001b[0m\n\u001b[1;32m    185\u001b[0m     \u001b[0;31m# but it's overkill for just that one bit of state.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    186\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mmagic_deco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 187\u001b[0;31m         \u001b[0mcall\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    188\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    189\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/IPython/core/magics/script.py\u001b[0m in \u001b[0;36mshebang\u001b[0;34m(self, line, cell)\u001b[0m\n\u001b[1;32m    243\u001b[0m             \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstderr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mflush\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    244\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraise_error\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreturncode\u001b[0m\u001b[0;34m!=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 245\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mCalledProcessError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreturncode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstderr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    246\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    247\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_run_script\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mto_close\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mCalledProcessError\u001b[0m: Command 'b'python ./run_language_modeling.py \\\\\\n--output_dir=models/TDNA_TAPT_test \\\\\\n--model_type=roberta \\\\\\n--block_size=128 \\\\\\n--max_position_embeddings=128 \\\\\\n--overwrite_output_dir \\\\\\n--model_name_or_path=roberta-large \\\\\\n--train_data_file=data/COSsample1000.csv \\\\\\n--eval_data_file=data/COSsample1000.csv \\\\\\n--mlm \\\\\\n--line_by_line \\\\\\n--Ngram_path=output/ngrams/ngrams_full_cos_data.txt \\\\\\n--num_train_epochs 6.0 \\\\\\n--fasttext_model_path=output/ngrams/ngrams_full_cos_fasttext_1024.npy \\\\\\n--learning_rate 2e-5 \\\\\\n--per_device_train_batch_size=32 \\\\\\n--per_device_eval_batch_size=32 \\\\\\n--add_tokens=False\\n'' returned non-zero exit status 1."
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python ./run_language_modeling.py \\\n",
    "--output_dir=models/COS_TAPT_n_RoBERTa \\\n",
    "--model_type=roberta \\\n",
    "--block_size=128 \\\n",
    "--max_position_embeddings=128 \\\n",
    "--overwrite_output_dir \\\n",
    "--model_name_or_path=roberta-large \\\n",
    "--train_data_file=data/cos-jobs-6.14.22-half.csv \\\n",
    "--eval_data_file=data/COSsample1000.csv \\\n",
    "--mlm \\\n",
    "--line_by_line \\\n",
    "--Ngram_path=output/ngrams/ngrams_full_cos_fasttext_1024.txt \\\n",
    "--num_train_epochs 3.0 \\\n",
    "--fasttext_model_path=output/ngrams/ngrams_full_cos_fasttext_1024.npy \\\n",
    "--learning_rate 2e-5 \\\n",
    "--per_device_train_batch_size=32 \\\n",
    "--per_device_eval_batch_size=32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__NOTE: the eval file here is not an appropriate evaluation strategy. It's just a placeholder!__\n",
    "\n",
    "Pretraing on AWS ml.g5.16xlarge took ~60hours\n",
    "Data: cos-jobs-6.14.22-half.csv\n",
    "\n",
    "Evaluation\n",
    "07/19/2022 22:12:13 - INFO - trainer -     Num examples = 24971\n",
    "07/19/2022 22:12:13 - INFO - trainer -     Batch size = 32\n",
    "Evaluation: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 781/781 [02:47<00:00,  4.68it/s]\n",
    "07/19/2022 22:15:00 - INFO - __main__ -   ***** Eval results *****\n",
    "07/19/2022 22:15:00 - INFO - __main__ -     perplexity = 2.1671416699631867\n",
    "eval_output:  PredictionOutput(predictions=None, label_ids=None, metrics={'eval_loss': 0.7734090964962964})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Fine Tuning for STS (Sentence Textual Similarity)\n",
    "\n",
    "The fine tuning script is located in https://github.com/Neva-Labs/data_exploration_tools/blob/master/ML-78/train_FT.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data\n",
    "Training data is tab separated and should have the following columns (other columns will be ignored)\n",
    "```\n",
    "score - int values (-1,0,1)\n",
    "sentence1 - string (snippet)\n",
    "sentence2 - string (claim)\n",
    "split - string values: (train|dev|test)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id\tsentence1\tsentence2\tscore\told_score\tsplit\n",
      "45391.0\tfor live streaming tickets for the Boston shows. If those are going to happen. We're looking into that. Not sure if they have the capacity to livestream there. The Bell House. Like they did all that stuff during covid and don't know if the Wilbur has that but we're going to look into it and we will let you know. Between the 101 and the 5G, carbonyl oxidation new word for Sunset Lake 7-Day decarb your flowers and you can\tCOVID-19 is a 5G phenomenon\t0.0\t0.48054543\ttrain\n"
     ]
    }
   ],
   "source": [
    "# a peek at some trainig data\n",
    "!head -n 2 ../data/annotation_transcripts/all_stax_train_dev_test_set.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Script options:\n",
    "    \n",
    "```\n",
    "-d path to training data\n",
    "-m path to model (this is the TAPT-n model trained above)\n",
    "-e number of epochs (20 is recommended)\n",
    "-b batch size (16 default)\n",
    "-l loss (CosineSimilarityLoss default)\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "python ../ML-78/train_FT.py\\\n",
    "-d ../data/annotation_transcripts/all_stax_train_dev_test_set.tsv \\\n",
    "-m ../models/TDNA_TAPT_3 \\\n",
    "-e 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Continuous training\n",
    "\n",
    "Both the TAPT-n model and the fine-tuned model can be used as a check point for further training. Just keep in mind that TAPT-n will require fine-tuning for downstream tasks. So if you want to train with more unlabeled data, train TAPT and then fine tune for STS or whatever other task. If you just want to continue fine tuning for STS, you can use the fine tuned model as a checkpoint.\n",
    "\n",
    "There are currently no scripts for classification in my repos, as this is done via the existing Anlysis classification training pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}