examples/lm-eval-overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Qw83KAePAhaS"
   },
   "source": [
    "# Releasing LM-Evaluation-Harness v0.4.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z7k2vq1iAdqr"
   },
   "source": [
    "With the vast amount of work done in the field today, it helps to have a tool that people can use easily to share their results and use to check others to ensure reported numbers are valid. The LM Evaluation Harness is one such tool the community has used extensively. We want to continue to support the community and with that in mind, we’re excited to announce a major update on the LM Evaluation Harness to further our goal for open and accessible AI research."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0gDoM0AJAvEc"
   },
   "source": [
    "Our refactor stems from our desires to make the following believed best practices easier to carry out.  \n",
    "\n",
    "1.   Never copy results from other papers\n",
    "2.   Always share your exact prompts\n",
    "3.   Always provide model outputs\n",
    "4.   Qualitatively review a small batch of outputs before running evaluation jobs at scale\n",
    "\n",
    "We also wanted to make the library a better experience to use and to contribute or design evaluations within. New features in the new release that serve this purpose include:\n",
    "\n",
    "1. Faster Evaluation Runtimes (accelerated data-parallel inference with HF Transformers + Accelerate, and commonly used or faster inference libraries such as vLLM and Llama-CPP)\n",
    "2. Easier addition and sharing of new tasks (YAML-based task config formats, allowing single-file sharing of custom tasks)\n",
    "3. More configurability, for more advanced workflows and easier operation with modifying prompts\n",
    "4. Better logging of data at runtime and post-hoc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nnwsOpjda_YW"
   },
   "source": [
    "In this notebook we will be going through a short tutorial on how things work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zAov81vTbL2K"
   },
   "source": [
    "## Install LM-Eval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8hiosGzq_qZg",
    "outputId": "6ab73e5e-1f54-417e-a388-07e0d870b132"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting git+https://github.com/EleutherAI/lm-evaluation-harness.git@big-refactor\n",
      "  Cloning https://github.com/EleutherAI/lm-evaluation-harness.git (to revision big-refactor) to /tmp/pip-req-build-tnssql5s\n",
      "  Running command git clone --filter=blob:none --quiet https://github.com/EleutherAI/lm-evaluation-harness.git /tmp/pip-req-build-tnssql5s\n",
      "  Running command git checkout -b big-refactor --track origin/big-refactor\n",
      "  Switched to a new branch 'big-refactor'\n",
      "  Branch 'big-refactor' set up to track remote branch 'big-refactor' from 'origin'.\n",
      "  Resolved https://github.com/EleutherAI/lm-evaluation-harness.git to commit 42f486ee49b65926a444cb0620870a39a5b4b0a8\n",
      "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
      "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
      "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
      "Collecting accelerate>=0.21.0 (from lm-eval==1.0.0)\n",
      "  Downloading accelerate-0.24.1-py3-none-any.whl (261 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m261.4/261.4 kB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting evaluate (from lm-eval==1.0.0)\n",
      "  Downloading evaluate-0.4.1-py3-none-any.whl (84 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m84.1/84.1 kB\u001b[0m \u001b[31m5.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting datasets>=2.0.0 (from lm-eval==1.0.0)\n",
      "  Downloading datasets-2.15.0-py3-none-any.whl (521 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m521.2/521.2 kB\u001b[0m \u001b[31m9.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting jsonlines (from lm-eval==1.0.0)\n",
      "  Downloading jsonlines-4.0.0-py3-none-any.whl (8.7 kB)\n",
      "Requirement already satisfied: numexpr in /usr/local/lib/python3.10/dist-packages (from lm-eval==1.0.0) (2.8.7)\n",
      "Collecting peft>=0.2.0 (from lm-eval==1.0.0)\n",
      "  Downloading peft-0.6.2-py3-none-any.whl (174 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m174.7/174.7 kB\u001b[0m \u001b[31m7.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting pybind11>=2.6.2 (from lm-eval==1.0.0)\n",
      "  Downloading pybind11-2.11.1-py3-none-any.whl (227 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m227.7/227.7 kB\u001b[0m \u001b[31m12.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting pytablewriter (from lm-eval==1.0.0)\n",
      "  Downloading pytablewriter-1.2.0-py3-none-any.whl (111 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.1/111.1 kB\u001b[0m \u001b[31m8.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting rouge-score>=0.0.4 (from lm-eval==1.0.0)\n",
      "  Downloading rouge_score-0.1.2.tar.gz (17 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "Collecting sacrebleu>=1.5.0 (from lm-eval==1.0.0)\n",
      "  Downloading sacrebleu-2.3.2-py3-none-any.whl (119 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m119.7/119.7 kB\u001b[0m \u001b[31m8.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: scikit-learn>=0.24.1 in /usr/local/lib/python3.10/dist-packages (from lm-eval==1.0.0) (1.2.2)\n",
      "Collecting sqlitedict (from lm-eval==1.0.0)\n",
      "  Downloading sqlitedict-2.1.0.tar.gz (21 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "Requirement already satisfied: torch>=1.8 in /usr/local/lib/python3.10/dist-packages (from lm-eval==1.0.0) (2.1.0+cu118)\n",
      "Collecting tqdm-multiprocess (from lm-eval==1.0.0)\n",
      "  Downloading tqdm_multiprocess-0.0.11-py3-none-any.whl (9.8 kB)\n",
      "Requirement already satisfied: transformers>=4.1 in /usr/local/lib/python3.10/dist-packages (from lm-eval==1.0.0) (4.35.2)\n",
      "Collecting zstandard (from lm-eval==1.0.0)\n",
      "  Downloading zstandard-0.22.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.4/5.4 MB\u001b[0m \u001b[31m29.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.21.0->lm-eval==1.0.0) (1.23.5)\n",
      "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.21.0->lm-eval==1.0.0) (23.2)\n",
      "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.21.0->lm-eval==1.0.0) (5.9.5)\n",
      "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.21.0->lm-eval==1.0.0) (6.0.1)\n",
      "Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.21.0->lm-eval==1.0.0) (0.19.4)\n",
      "Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.0.0->lm-eval==1.0.0) (9.0.0)\n",
      "Collecting pyarrow-hotfix (from datasets>=2.0.0->lm-eval==1.0.0)\n",
      "  Downloading pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB)\n",
      "Collecting dill<0.3.8,>=0.3.0 (from datasets>=2.0.0->lm-eval==1.0.0)\n",
      "  Downloading dill-0.3.7-py3-none-any.whl (115 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m115.3/115.3 kB\u001b[0m \u001b[31m14.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets>=2.0.0->lm-eval==1.0.0) (1.5.3)\n",
      "Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.0.0->lm-eval==1.0.0) (2.31.0)\n",
      "Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.0.0->lm-eval==1.0.0) (4.66.1)\n",
      "Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets>=2.0.0->lm-eval==1.0.0) (3.4.1)\n",
      "Collecting multiprocess (from datasets>=2.0.0->lm-eval==1.0.0)\n",
      "  Downloading multiprocess-0.70.15-py310-none-any.whl (134 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m19.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: fsspec[http]<=2023.10.0,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.0.0->lm-eval==1.0.0) (2023.6.0)\n",
      "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets>=2.0.0->lm-eval==1.0.0) (3.8.6)\n",
      "Collecting responses<0.19 (from evaluate->lm-eval==1.0.0)\n",
      "  Downloading responses-0.18.0-py3-none-any.whl (38 kB)\n",
      "Requirement already satisfied: safetensors in /usr/local/lib/python3.10/dist-packages (from peft>=0.2.0->lm-eval==1.0.0) (0.4.0)\n",
      "Requirement already satisfied: absl-py in /usr/local/lib/python3.10/dist-packages (from rouge-score>=0.0.4->lm-eval==1.0.0) (1.4.0)\n",
      "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from rouge-score>=0.0.4->lm-eval==1.0.0) (3.8.1)\n",
      "Requirement already satisfied: six>=1.14.0 in /usr/local/lib/python3.10/dist-packages (from rouge-score>=0.0.4->lm-eval==1.0.0) (1.16.0)\n",
      "Collecting portalocker (from sacrebleu>=1.5.0->lm-eval==1.0.0)\n",
      "  Downloading portalocker-2.8.2-py3-none-any.whl (17 kB)\n",
      "Requirement already satisfied: regex in /usr/local/lib/python3.10/dist-packages (from sacrebleu>=1.5.0->lm-eval==1.0.0) (2023.6.3)\n",
      "Requirement already satisfied: tabulate>=0.8.9 in /usr/local/lib/python3.10/dist-packages (from sacrebleu>=1.5.0->lm-eval==1.0.0) (0.9.0)\n",
      "Collecting colorama (from sacrebleu>=1.5.0->lm-eval==1.0.0)\n",
      "  Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n",
      "Requirement already satisfied: lxml in /usr/local/lib/python3.10/dist-packages (from sacrebleu>=1.5.0->lm-eval==1.0.0) (4.9.3)\n",
      "Requirement already satisfied: scipy>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm-eval==1.0.0) (1.11.3)\n",
      "Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm-eval==1.0.0) (1.3.2)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm-eval==1.0.0) (3.2.0)\n",
      "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.8->lm-eval==1.0.0) (3.13.1)\n",
      "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch>=1.8->lm-eval==1.0.0) (4.5.0)\n",
      "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.8->lm-eval==1.0.0) (1.12)\n",
      "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.8->lm-eval==1.0.0) (3.2.1)\n",
      "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.8->lm-eval==1.0.0) (3.1.2)\n",
      "Requirement already satisfied: triton==2.1.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.8->lm-eval==1.0.0) (2.1.0)\n",
      "Requirement already satisfied: tokenizers<0.19,>=0.14 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.1->lm-eval==1.0.0) (0.15.0)\n",
      "Requirement already satisfied: attrs>=19.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonlines->lm-eval==1.0.0) (23.1.0)\n",
      "Requirement already satisfied: setuptools>=38.3.0 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm-eval==1.0.0) (67.7.2)\n",
      "Collecting DataProperty<2,>=1.0.1 (from pytablewriter->lm-eval==1.0.0)\n",
      "  Downloading DataProperty-1.0.1-py3-none-any.whl (27 kB)\n",
      "Collecting mbstrdecoder<2,>=1.0.0 (from pytablewriter->lm-eval==1.0.0)\n",
      "  Downloading mbstrdecoder-1.1.3-py3-none-any.whl (7.8 kB)\n",
      "Collecting pathvalidate<4,>=2.3.0 (from pytablewriter->lm-eval==1.0.0)\n",
      "  Downloading pathvalidate-3.2.0-py3-none-any.whl (23 kB)\n",
      "Collecting tabledata<2,>=1.3.1 (from pytablewriter->lm-eval==1.0.0)\n",
      "  Downloading tabledata-1.3.3-py3-none-any.whl (11 kB)\n",
      "Collecting tcolorpy<1,>=0.0.5 (from pytablewriter->lm-eval==1.0.0)\n",
      "  Downloading tcolorpy-0.1.4-py3-none-any.whl (7.9 kB)\n",
      "Collecting typepy[datetime]<2,>=1.3.2 (from pytablewriter->lm-eval==1.0.0)\n",
      "  Downloading typepy-1.3.2-py3-none-any.whl (31 kB)\n",
      "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval==1.0.0) (3.3.2)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval==1.0.0) (6.0.4)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval==1.0.0) (4.0.3)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval==1.0.0) (1.9.2)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval==1.0.0) (1.4.0)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->lm-eval==1.0.0) (1.3.1)\n",
      "Requirement already satisfied: chardet<6,>=3.0.4 in /usr/local/lib/python3.10/dist-packages (from mbstrdecoder<2,>=1.0.0->pytablewriter->lm-eval==1.0.0) (5.2.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets>=2.0.0->lm-eval==1.0.0) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets>=2.0.0->lm-eval==1.0.0) (2.0.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets>=2.0.0->lm-eval==1.0.0) (2023.7.22)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.8.0 in /usr/local/lib/python3.10/dist-packages (from typepy[datetime]<2,>=1.3.2->pytablewriter->lm-eval==1.0.0) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2018.9 in /usr/local/lib/python3.10/dist-packages (from typepy[datetime]<2,>=1.3.2->pytablewriter->lm-eval==1.0.0) (2023.3.post1)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.8->lm-eval==1.0.0) (2.1.3)\n",
      "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk->rouge-score>=0.0.4->lm-eval==1.0.0) (8.1.7)\n",
      "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.8->lm-eval==1.0.0) (1.3.0)\n",
      "Building wheels for collected packages: lm-eval, rouge-score, sqlitedict\n",
      "  Building wheel for lm-eval (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for lm-eval: filename=lm_eval-1.0.0-py3-none-any.whl size=994254 sha256=88356155b19f2891981ecef948326ad6ce8ca40a6009378410ec20d0e225995a\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-9v6ye7h3/wheels/17/01/26/599c0779e9858a70a73fa8a306699b5b9a868f820c225457b0\n",
      "  Building wheel for rouge-score (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for rouge-score: filename=rouge_score-0.1.2-py3-none-any.whl size=24933 sha256=6bb0d44e4881972c43ce194e7cb65233d309758cb15f0dec54590d3d2efcfc36\n",
      "  Stored in directory: /root/.cache/pip/wheels/5f/dd/89/461065a73be61a532ff8599a28e9beef17985c9e9c31e541b4\n",
      "  Building wheel for sqlitedict (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for sqlitedict: filename=sqlitedict-2.1.0-py3-none-any.whl size=16863 sha256=5747f7dd73ddf3d8fbcebf51b5e4f718fabe1e94bccdf16d2f22a2e65ee7fdf4\n",
      "  Stored in directory: /root/.cache/pip/wheels/79/d6/e7/304e0e6cb2221022c26d8161f7c23cd4f259a9e41e8bbcfabd\n",
      "Successfully built lm-eval rouge-score sqlitedict\n",
      "Installing collected packages: sqlitedict, zstandard, tcolorpy, pybind11, pyarrow-hotfix, portalocker, pathvalidate, mbstrdecoder, jsonlines, dill, colorama, typepy, tqdm-multiprocess, sacrebleu, rouge-score, responses, multiprocess, accelerate, datasets, DataProperty, tabledata, peft, evaluate, pytablewriter, lm-eval\n",
      "Successfully installed DataProperty-1.0.1 accelerate-0.24.1 colorama-0.4.6 datasets-2.15.0 dill-0.3.7 evaluate-0.4.1 jsonlines-4.0.0 lm-eval-1.0.0 mbstrdecoder-1.1.3 multiprocess-0.70.15 pathvalidate-3.2.0 peft-0.6.2 portalocker-2.8.2 pyarrow-hotfix-0.6 pybind11-2.11.1 pytablewriter-1.2.0 responses-0.18.0 rouge-score-0.1.2 sacrebleu-2.3.2 sqlitedict-2.1.0 tabledata-1.3.3 tcolorpy-0.1.4 tqdm-multiprocess-0.0.11 typepy-1.3.2 zstandard-0.22.0\n"
     ]
    }
   ],
   "source": [
    "# Install LM-Eval\n",
    "!pip install git+https://github.com/EleutherAI/lm-evaluation-harness.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0,
     "referenced_widgets": [
      "a1d3a8aa016544a78e8821c8f6199e06",
      "f61ed33fad754146bdd2ac9db1ba1c48",
      "bfa0af6aeff344c6845e1080a878e92e",
      "fd1ad9e0367d4004aae853b91c3a7617",
      "6b2d90209ec14230b3d58a74ac9b83bf",
      "a73f357065d34d7baf0453ae4a8d75e2",
      "46f521b73fd943c081c648fd873ebc0a",
      "7c5689bc13684db8a22681f41863dddd",
      "48763b6233374554ae76035c0483066f",
      "4986a21eb560448fa79f4b25cde48951",
      "aed3acd2f2d74003b44079c333a0698e"
     ]
    },
    "id": "uyO5MaKkZyah",
    "outputId": "d46e8096-5086-4e49-967e-ea33d4a2a335"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a1d3a8aa016544a78e8821c8f6199e06",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading builder script:   0%|          | 0.00/5.67k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8rfUeX6n_wkK"
   },
   "source": [
    "## Create new evaluation tasks with config-based tasks\n",
    "\n",
    "Even within the same task, many works have reported numbers based on different choices of evaluation. Some report on the test sets, validation sets, or even subset of the training sets. Others have specialized prompts and verbalizers. We introduce YAMLs to allow users to easily make different variations. By leveraging the YAML configs to configure evaluations, the refactored LM-Eval takes the methods of the `Task` object and makes them configurable by setting the appropriate attributes in the config file. There, users can set the tasks they want by setting the name of the HF dataset (local tasks are also possible), the dataset splits used, and much more. Key configurations relating to prompting, such as `doc_to_text`, previously implemented as a method of the same name, are now configurable with jinja2 to allow high-level scripting to transform a HF dataset to text string as input to the model.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HYFUhhfOSJKe"
   },
   "source": [
    "A core-feature to LM-Eval is to configure tasks with YAML configs. With configs, you can fill preset fields to easily set up a task.\n",
    "\n",
    "Here, we write a demo YAML config for a multiple-choice evaluation of BoolQ:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "bg3dGROW-V39"
   },
   "outputs": [],
   "source": [
    "YAML_boolq_string = \"\"\"\n",
    "task: demo_boolq\n",
    "dataset_path: super_glue\n",
    "dataset_name: boolq\n",
    "output_type: multiple_choice\n",
    "training_split: train\n",
    "validation_split: validation\n",
    "doc_to_text: \"{{passage}}\\nQuestion: {{question}}?\\nAnswer:\"\n",
    "doc_to_target: label\n",
    "doc_to_choice: [\"no\", \"yes\"]\n",
    "should_decontaminate: true\n",
    "doc_to_decontamination_query: passage\n",
    "metric_list:\n",
    "  - metric: acc\n",
    "\"\"\"\n",
    "with open(\"boolq.yaml\", \"w\") as f:\n",
    "    f.write(YAML_boolq_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can now run evaluation on this task, by pointing to the config file we've just created:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "LOUHK7PtQfq4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-11-29:11:54:55,156 INFO     [utils.py:160] NumExpr defaulting to 2 threads.\n",
      "2023-11-29 11:54:55.942051: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2023-11-29 11:54:55.942108: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2023-11-29 11:54:55.942142: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2023-11-29 11:54:57.066802: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
      "2023-11-29:11:55:00,954 INFO     [__main__.py:132] Verbosity set to INFO\n",
      "2023-11-29:11:55:11,038 WARNING  [__main__.py:138]  --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.\n",
      "2023-11-29:11:55:11,038 INFO     [__main__.py:143] Including path: ./\n",
      "2023-11-29:11:55:11,046 INFO     [__main__.py:205] Selected Tasks: ['demo_boolq']\n",
      "2023-11-29:11:55:11,047 WARNING  [evaluator.py:93] generation_kwargs specified through cli, these settings will be used over set parameters in yaml tasks.\n",
      "2023-11-29:11:55:11,110 INFO     [huggingface.py:120] Using device 'cuda'\n",
      "config.json: 100% 571/571 [00:00<00:00, 2.87MB/s]\n",
      "model.safetensors: 100% 5.68G/5.68G [00:32<00:00, 173MB/s]\n",
      "tokenizer_config.json: 100% 396/396 [00:00<00:00, 2.06MB/s]\n",
      "tokenizer.json: 100% 2.11M/2.11M [00:00<00:00, 11.6MB/s]\n",
      "special_tokens_map.json: 100% 99.0/99.0 [00:00<00:00, 555kB/s]\n",
      "2023-11-29:11:56:18,658 WARNING  [task.py:614] [Task: demo_boolq] metric acc is defined, but aggregation is not. using default aggregation=mean\n",
      "2023-11-29:11:56:18,658 WARNING  [task.py:626] [Task: demo_boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True\n",
      "Downloading builder script: 100% 30.7k/30.7k [00:00<00:00, 59.0MB/s]\n",
      "Downloading metadata: 100% 38.7k/38.7k [00:00<00:00, 651kB/s]\n",
      "Downloading readme: 100% 14.8k/14.8k [00:00<00:00, 37.3MB/s]\n",
      "Downloading data: 100% 4.12M/4.12M [00:00<00:00, 55.1MB/s]\n",
      "Generating train split: 100% 9427/9427 [00:00<00:00, 15630.89 examples/s]\n",
      "Generating validation split: 100% 3270/3270 [00:00<00:00, 20002.56 examples/s]\n",
      "Generating test split: 100% 3245/3245 [00:00<00:00, 20866.19 examples/s]\n",
      "2023-11-29:11:56:22,315 INFO     [task.py:355] Building contexts for task on rank 0...\n",
      "2023-11-29:11:56:22,322 INFO     [evaluator.py:319] Running loglikelihood requests\n",
      "100% 20/20 [00:04<00:00,  4.37it/s]\n",
      "fatal: not a git repository (or any of the parent directories): .git\n",
      "hf (pretrained=EleutherAI/pythia-2.8b), gen_kwargs: (), limit: 10.0, num_fewshot: None, batch_size: 1\n",
      "|  Tasks   |Version|Filter|n-shot|Metric|Value|   |Stderr|\n",
      "|----------|-------|------|-----:|------|----:|---|-----:|\n",
      "|demo_boolq|Yaml   |none  |     0|acc   |    1|±  |     0|\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!lm_eval \\\n",
    "    --model hf \\\n",
    "    --model_args pretrained=EleutherAI/pythia-2.8b \\\n",
    "    --include_path ./ \\\n",
    "    --tasks demo_boolq \\\n",
    "    --limit 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LOUHK7PtQfq4"
   },
   "source": [
    "Often, tasks are part of a larger group used to measure different capabilities. The dynamism of the field today means new dimensions of evaluation can come about which would mix and match new and older tasks alike. In LM-Eval, We can also group tasks and call that the group name to evaluate on a set of tasks easily. In this instance, let's evaluate the tag `yes_or_no_tasks` which comprise of the tasks `demo_boolq` and `demo_cola`; tasks which are multiple choice tasks with options `yes` and `no` as the name suggests.\n",
    "\n",
    "<!-- making new groups is easier than ever, allowing user to work bottom-up by makiing individual tasks and linking them to a group or Top-Down, making a new group by listing existing tasks.\n",
    "\n",
    "We also show the aggregate across samples besides only showing the aggregation between subtasks. This may come in handy when certain groups want to be aggregated as a single task. -->\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "fthNg3ywO-kA"
   },
   "outputs": [],
   "source": [
    "YAML_cola_string = \"\"\"\n",
    "tag: yes_or_no_tasks\n",
    "task: demo_cola\n",
    "dataset_path: glue\n",
    "dataset_name: cola\n",
    "output_type: multiple_choice\n",
    "training_split: train\n",
    "validation_split: validation\n",
    "doc_to_text: \"{{sentence}}\\nQuestion: Does this sentence make sense?\\nAnswer:\"\n",
    "doc_to_target: label\n",
    "doc_to_choice: [\"no\", \"yes\"]\n",
    "should_decontaminate: true\n",
    "doc_to_decontamination_query: sentence\n",
    "metric_list:\n",
    "  - metric: acc\n",
    "\"\"\"\n",
    "with open(\"cola.yaml\", \"w\") as f:\n",
    "    f.write(YAML_cola_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "XceRKCuuDtbn"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-11-29:11:56:33,016 INFO     [utils.py:160] NumExpr defaulting to 2 threads.\n",
      "2023-11-29 11:56:33.852995: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2023-11-29 11:56:33.853050: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2023-11-29 11:56:33.853087: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2023-11-29 11:56:35.129047: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
      "2023-11-29:11:56:38,546 INFO     [__main__.py:132] Verbosity set to INFO\n",
      "2023-11-29:11:56:47,509 WARNING  [__main__.py:138]  --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.\n",
      "2023-11-29:11:56:47,509 INFO     [__main__.py:143] Including path: ./\n",
      "2023-11-29:11:56:47,517 INFO     [__main__.py:205] Selected Tasks: ['yes_or_no_tasks']\n",
      "2023-11-29:11:56:47,520 WARNING  [evaluator.py:93] generation_kwargs specified through cli, these settings will be used over set parameters in yaml tasks.\n",
      "2023-11-29:11:56:47,550 INFO     [huggingface.py:120] Using device 'cuda'\n",
      "2023-11-29:11:57:08,743 WARNING  [task.py:614] [Task: demo_cola] metric acc is defined, but aggregation is not. using default aggregation=mean\n",
      "2023-11-29:11:57:08,743 WARNING  [task.py:626] [Task: demo_cola] metric acc is defined, but higher_is_better is not. using default higher_is_better=True\n",
      "Downloading builder script: 100% 28.8k/28.8k [00:00<00:00, 52.7MB/s]\n",
      "Downloading metadata: 100% 28.7k/28.7k [00:00<00:00, 51.9MB/s]\n",
      "Downloading readme: 100% 27.9k/27.9k [00:00<00:00, 48.0MB/s]\n",
      "Downloading data: 100% 377k/377k [00:00<00:00, 12.0MB/s]\n",
      "Generating train split: 100% 8551/8551 [00:00<00:00, 19744.58 examples/s]\n",
      "Generating validation split: 100% 1043/1043 [00:00<00:00, 27057.01 examples/s]\n",
      "Generating test split: 100% 1063/1063 [00:00<00:00, 22705.17 examples/s]\n",
      "2023-11-29:11:57:11,698 INFO     [task.py:355] Building contexts for task on rank 0...\n",
      "2023-11-29:11:57:11,704 INFO     [evaluator.py:319] Running loglikelihood requests\n",
      "100% 20/20 [00:03<00:00,  5.15it/s]\n",
      "fatal: not a git repository (or any of the parent directories): .git\n",
      "hf (pretrained=EleutherAI/pythia-2.8b), gen_kwargs: (), limit: 10.0, num_fewshot: None, batch_size: 1\n",
      "|     Tasks     |Version|Filter|n-shot|Metric|Value|   |Stderr|\n",
      "|---------------|-------|------|-----:|------|----:|---|-----:|\n",
      "|yes_or_no_tasks|N/A    |none  |     0|acc   |  0.7|±  |0.1528|\n",
      "| - demo_cola   |Yaml   |none  |     0|acc   |  0.7|±  |0.1528|\n",
      "\n",
      "|    Groups     |Version|Filter|n-shot|Metric|Value|   |Stderr|\n",
      "|---------------|-------|------|-----:|------|----:|---|-----:|\n",
      "|yes_or_no_tasks|N/A    |none  |     0|acc   |  0.7|±  |0.1528|\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# !accelerate launch --no_python\n",
    "!lm_eval \\\n",
    "    --model hf \\\n",
    "    --model_args pretrained=EleutherAI/pythia-2.8b \\\n",
    "    --include_path ./ \\\n",
    "    --tasks yes_or_no_tasks \\\n",
    "    --limit 10 \\\n",
    "    --output output/yes_or_no_tasks/ \\\n",
    "    --log_samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XceRKCuuDtbn"
   },
   "source": [
    "## Edit Prompt Templates Quickly\n",
    "\n",
    "The following is a yaml made to evaluate the specific subtask of `high_school_geography` from MMLU. It uses the standard prompt where the we choose the letters from the options with most likelihood as the model's prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "GTFvdt9kSlBG"
   },
   "outputs": [],
   "source": [
    "YAML_mmlu_geo_string = \"\"\"\n",
    "task: demo_mmlu_high_school_geography\n",
    "dataset_path: cais/mmlu\n",
    "dataset_name: high_school_geography\n",
    "description: \"The following are multiple choice questions (with answers) about high school geography.\\n\\n\"\n",
    "test_split: test\n",
    "fewshot_split: dev\n",
    "fewshot_config:\n",
    "  sampler: first_n\n",
    "output_type: multiple_choice\n",
    "doc_to_text: \"{{question.strip()}}\\nA. {{choices[0]}}\\nB. {{choices[1]}}\\nC. {{choices[2]}}\\nD. {{choices[3]}}\\nAnswer:\"\n",
    "doc_to_choice: [\"A\", \"B\", \"C\", \"D\"]\n",
    "doc_to_target: answer\n",
    "metric_list:\n",
    "  - metric: acc\n",
    "    aggregation: mean\n",
    "    higher_is_better: true\n",
    "  - metric: acc_norm\n",
    "    aggregation: mean\n",
    "    higher_is_better: true\n",
    "\"\"\"\n",
    "with open(\"mmlu_high_school_geography.yaml\", \"w\") as f:\n",
    "    f.write(YAML_mmlu_geo_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "jyKOfCsKb-xy"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-11-29:11:57:23,598 INFO     [utils.py:160] NumExpr defaulting to 2 threads.\n",
      "2023-11-29 11:57:24.719750: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2023-11-29 11:57:24.719806: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2023-11-29 11:57:24.719847: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2023-11-29 11:57:26.656125: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
      "2023-11-29:11:57:31,563 INFO     [__main__.py:132] Verbosity set to INFO\n",
      "2023-11-29:11:57:40,541 WARNING  [__main__.py:138]  --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.\n",
      "2023-11-29:11:57:40,541 INFO     [__main__.py:143] Including path: ./\n",
      "2023-11-29:11:57:40,558 INFO     [__main__.py:205] Selected Tasks: ['demo_mmlu_high_school_geography']\n",
      "2023-11-29:11:57:40,559 WARNING  [evaluator.py:93] generation_kwargs specified through cli, these settings will be used over set parameters in yaml tasks.\n",
      "2023-11-29:11:57:40,589 INFO     [huggingface.py:120] Using device 'cuda'\n",
      "Downloading builder script: 100% 5.84k/5.84k [00:00<00:00, 17.7MB/s]\n",
      "Downloading metadata: 100% 106k/106k [00:00<00:00, 892kB/s] \n",
      "Downloading readme: 100% 39.7k/39.7k [00:00<00:00, 631kB/s]\n",
      "Downloading data: 100% 166M/166M [00:01<00:00, 89.0MB/s]\n",
      "Generating auxiliary_train split: 100% 99842/99842 [00:07<00:00, 12536.83 examples/s]\n",
      "Generating test split: 100% 198/198 [00:00<00:00, 1439.20 examples/s]\n",
      "Generating validation split: 100% 22/22 [00:00<00:00, 4181.76 examples/s]\n",
      "Generating dev split: 100% 5/5 [00:00<00:00, 36.25 examples/s]\n",
      "2023-11-29:11:58:09,798 INFO     [task.py:355] Building contexts for task on rank 0...\n",
      "2023-11-29:11:58:09,822 INFO     [evaluator.py:319] Running loglikelihood requests\n",
      "100% 40/40 [00:05<00:00,  7.86it/s]\n",
      "fatal: not a git repository (or any of the parent directories): .git\n",
      "hf (pretrained=EleutherAI/pythia-2.8b), gen_kwargs: (), limit: 10.0, num_fewshot: None, batch_size: 1\n",
      "|             Tasks             |Version|Filter|n-shot| Metric |Value|   |Stderr|\n",
      "|-------------------------------|-------|------|-----:|--------|----:|---|-----:|\n",
      "|demo_mmlu_high_school_geography|Yaml   |none  |     0|acc     |  0.3|±  |0.1528|\n",
      "|                               |       |none  |     0|acc_norm|  0.3|±  |0.1528|\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# !accelerate launch --no_python\n",
    "!lm_eval \\\n",
    "    --model hf \\\n",
    "    --model_args pretrained=EleutherAI/pythia-2.8b \\\n",
    "    --include_path ./ \\\n",
    "    --tasks demo_mmlu_high_school_geography \\\n",
    "    --limit 10 \\\n",
    "    --output output/mmlu_high_school_geography/ \\\n",
    "    --log_samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jyKOfCsKb-xy"
   },
   "source": [
    "We could also evaluate this task in a different way. For example, instead of observing the loglikelihood of the letters, we can instead evaluate on the choices themselves as the continuation. This is done by simply changing `doc_to_choice` from a list of letters to the corresponding `choices` field from the HF dataset. We write `\"{{choices}}\"` so that the string field is interpreted as jinja string that acquires the list from the HF dataset directly.\n",
    "\n",
    "Another convenient feature here is since we're only modifying the `doc_to_choice` and the rest of config is the same as the task above, we can use the above configuration as a template by using `include: mmlu_high_school_geography.yaml` to load the config from that file. We'll need to add a unique task name as to not colide with the existing yaml config we're including. For this case we'll simply name this one `mmlu_high_school_geography_continuation`. `doc_to_text` is added here just for sake of clarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "lqElwU54TaK-"
   },
   "outputs": [],
   "source": [
    "YAML_mmlu_geo_string = \"\"\"\n",
    "include: mmlu_high_school_geography.yaml\n",
    "task: demo_mmlu_high_school_geography_continuation\n",
    "doc_to_text: \"{{question.strip()}}\\nA. {{choices[0]}}\\nB. {{choices[1]}}\\nC. {{choices[2]}}\\nD. {{choices[3]}}\\nAnswer:\"\n",
    "doc_to_choice: \"{{choices}}\"\n",
    "\"\"\"\n",
    "with open(\"mmlu_high_school_geography_continuation.yaml\", \"w\") as f:\n",
    "    f.write(YAML_mmlu_geo_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "-_CVnDirdy7j"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-11-29:11:58:21,284 INFO     [utils.py:160] NumExpr defaulting to 2 threads.\n",
      "2023-11-29 11:58:22.850159: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2023-11-29 11:58:22.850219: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2023-11-29 11:58:22.850254: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2023-11-29 11:58:24.948103: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
      "2023-11-29:11:58:28,460 INFO     [__main__.py:132] Verbosity set to INFO\n",
      "2023-11-29:11:58:37,935 WARNING  [__main__.py:138]  --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.\n",
      "2023-11-29:11:58:37,935 INFO     [__main__.py:143] Including path: ./\n",
      "2023-11-29:11:58:37,969 INFO     [__main__.py:205] Selected Tasks: ['demo_mmlu_high_school_geography_continuation']\n",
      "2023-11-29:11:58:37,972 WARNING  [evaluator.py:93] generation_kwargs specified through cli, these settings will be used over set parameters in yaml tasks.\n",
      "2023-11-29:11:58:38,008 INFO     [huggingface.py:120] Using device 'cuda'\n",
      "2023-11-29:11:58:59,758 INFO     [task.py:355] Building contexts for task on rank 0...\n",
      "2023-11-29:11:58:59,777 INFO     [evaluator.py:319] Running loglikelihood requests\n",
      "100% 40/40 [00:02<00:00, 16.23it/s]\n",
      "fatal: not a git repository (or any of the parent directories): .git\n",
      "hf (pretrained=EleutherAI/pythia-2.8b), gen_kwargs: (), limit: 10.0, num_fewshot: None, batch_size: 1\n",
      "|                   Tasks                    |Version|Filter|n-shot| Metric |Value|   |Stderr|\n",
      "|--------------------------------------------|-------|------|-----:|--------|----:|---|-----:|\n",
      "|demo_mmlu_high_school_geography_continuation|Yaml   |none  |     0|acc     |  0.1|±  |0.1000|\n",
      "|                                            |       |none  |     0|acc_norm|  0.2|±  |0.1333|\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# !accelerate launch --no_python\n",
    "!lm_eval \\\n",
    "    --model hf \\\n",
    "    --model_args pretrained=EleutherAI/pythia-2.8b \\\n",
    "    --include_path ./ \\\n",
    "    --tasks demo_mmlu_high_school_geography_continuation \\\n",
    "    --limit 10 \\\n",
    "    --output output/mmlu_high_school_geography_continuation/ \\\n",
    "    --log_samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-_CVnDirdy7j"
   },
   "source": [
    "If we take a look at the samples, we can see that it is in fact evaluating the continuation based on the choices rather than the letters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "duBDqC6PAdjL"
   },
   "outputs": [
    {
     "data": {
      "application/javascript": "\n      ((filepath) => {{\n        if (!google.colab.kernel.accessAllowed) {{\n          return;\n        }}\n        google.colab.files.view(filepath);\n      }})(\"/content/output/mmlu_high_school_geography_continuation/pretrained__EleutherAI__pythia-2.8b_demo_mmlu_high_school_geography_continuation.jsonl\")",
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from google.colab import files\n",
    "\n",
    "\n",
    "files.view(\n",
    "    \"output/mmlu_high_school_geography_continuation/pretrained__EleutherAI__pythia-2.8b_demo_mmlu_high_school_geography_continuation.jsonl\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6p0-KPwAgK5j"
   },
   "source": [
    "## Closer Look at YAML Fields\n",
    "\n",
    "To prepare a task we can simply fill in a YAML config with the relevant information.\n",
    "\n",
    "`output_type`\n",
    "The current provided evaluation types comprise of the following:\n",
    "1.   `loglikelihood`: Evaluates the loglikelihood of a continuation, conditioned on some input string.\n",
    "2.   `loglikelihood_rolling`: evaluate the loglikelihood of producing a string, conditioned on the empty string. (Used for perplexity evaluations)\n",
    "3.   `multiple_choice`: Evaluates loglikelihood among the a number of choices predicted by the model.\n",
    "4.   `greedy_until`: Model outputs greedy generation (can be configured to to use beam search and other generation-related parameters)\n",
    "\n",
    "The core prompt revolves around 3 fields.\n",
    "1. `doc_to_text`: Denotes the prompt template that will be used as input to the model.\n",
    "2. `doc_to_choice`: Available choices that will be used as continuation for the model. This is used when the `output_type` is `multiple_choice`, and otherwise can be left as `None`.\n",
    "3. `doc_to_target`: When `output_type` is `multiple_choice`, this can be an index that corresponds to the correct answer, or the answer string itself (must be a subset of `doc_to_choice`). For other tasks, this is expected to be a string. You can fill this field with a feature name from the HF dataset so long as the resulting feature follows the conditioned described.\n",
    "\n",
    "These three fields can be expressed as strings, column names from the source dataset, or as Jinja2 templates that can use fields from the source dataset as variables.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6p0-KPwAgK5j"
   },
   "source": [
    "## What if Jinja is not Sufficient?\n",
    "\n",
    "There can be times where the Jinja2 templating language is not enough to make the prompt we had in mind. There are a few ways to circumvent this limitation:\n",
    "\n",
    "1. Use `!function` operator for the prompt-related fields to pass a python function that takes as input the dataset row, and will output the prompt template component.\n",
    "2. Perform a transformation on the dataset beforehand."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we show an example of using `!function` to create `doc_to_text` from a python function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DYZ5c0JhR1lJ",
    "outputId": "ca945235-fb9e-4f17-8bfa-78e7d6ec1490"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-11-29:11:59:08,312 INFO     [utils.py:160] NumExpr defaulting to 2 threads.\n",
      "2023-11-29 11:59:09.348327: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2023-11-29 11:59:09.348387: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2023-11-29 11:59:09.348421: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2023-11-29 11:59:10.573752: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
      "2023-11-29:11:59:14,044 INFO     [__main__.py:132] Verbosity set to INFO\n",
      "2023-11-29:11:59:23,654 WARNING  [__main__.py:138]  --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.\n",
      "2023-11-29:11:59:23,654 INFO     [__main__.py:143] Including path: ./\n",
      "2023-11-29:11:59:23,678 INFO     [__main__.py:205] Selected Tasks: ['demo_mmlu_high_school_geography_function_prompt']\n",
      "2023-11-29:11:59:23,679 WARNING  [evaluator.py:93] generation_kwargs specified through cli, these settings will be used over set parameters in yaml tasks.\n",
      "2023-11-29:11:59:23,708 INFO     [huggingface.py:120] Using device 'cuda'\n",
      "2023-11-29:11:59:44,516 INFO     [task.py:355] Building contexts for task on rank 0...\n",
      "2023-11-29:11:59:44,524 INFO     [evaluator.py:319] Running loglikelihood requests\n",
      "100% 40/40 [00:02<00:00, 15.41it/s]\n",
      "fatal: not a git repository (or any of the parent directories): .git\n",
      "hf (pretrained=EleutherAI/pythia-2.8b), gen_kwargs: (), limit: 10.0, num_fewshot: None, batch_size: 1\n",
      "|                     Tasks                     |Version|Filter|n-shot| Metric |Value|   |Stderr|\n",
      "|-----------------------------------------------|-------|------|-----:|--------|----:|---|-----:|\n",
      "|demo_mmlu_high_school_geography_function_prompt|Yaml   |none  |     0|acc     |  0.1|±  |0.1000|\n",
      "|                                               |       |none  |     0|acc_norm|  0.2|±  |0.1333|\n",
      "\n"
     ]
    }
   ],
   "source": [
    "YAML_mmlu_geo_string = \"\"\"\n",
    "include: mmlu_high_school_geography.yaml\n",
    "task: demo_mmlu_high_school_geography_function_prompt\n",
    "doc_to_text: !function utils.doc_to_text\n",
    "doc_to_choice: \"{{choices}}\"\n",
    "\"\"\"\n",
    "with open(\"demo_mmlu_high_school_geography_function_prompt.yaml\", \"w\") as f:\n",
    "    f.write(YAML_mmlu_geo_string)\n",
    "\n",
    "DOC_TO_TEXT = \"\"\"\n",
    "def doc_to_text(x):\n",
    "    question = x[\"question\"].strip()\n",
    "    choices = x[\"choices\"]\n",
    "    option_a = choices[0]\n",
    "    option_b = choices[1]\n",
    "    option_c = choices[2]\n",
    "    option_d = choices[3]\n",
    "    return f\"{question}\\\\nA. {option_a}\\\\nB. {option_b}\\\\nC. {option_c}\\\\nD. {option_d}\\\\nAnswer:\"\n",
    "\"\"\"\n",
    "with open(\"utils.py\", \"w\") as f:\n",
    "    f.write(DOC_TO_TEXT)\n",
    "\n",
    "!lm_eval \\\n",
    "    --model hf \\\n",
    "    --model_args pretrained=EleutherAI/pythia-2.8b \\\n",
    "    --include_path ./ \\\n",
    "    --tasks demo_mmlu_high_school_geography_function_prompt \\\n",
    "    --limit 10 \\\n",
    "    --output output/demo_mmlu_high_school_geography_function_prompt/ \\\n",
    "    --log_samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll also show how to do this via preprocessing the dataset as necessary using the `process_docs` config field:\n",
    "\n",
    "We will write a function that will modify each document in our evaluation dataset's split to add a field that is suitable for us to use in `doc_to_text`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "YAML_mmlu_geo_string = \"\"\"\n",
    "include: mmlu_high_school_geography.yaml\n",
    "task: demo_mmlu_high_school_geography_function_prompt_2\n",
    "process_docs: !function utils_process_docs.process_docs\n",
    "doc_to_text: \"{{input}}\"\n",
    "doc_to_choice: \"{{choices}}\"\n",
    "\"\"\"\n",
    "with open(\"demo_mmlu_high_school_geography_process_docs.yaml\", \"w\") as f:\n",
    "    f.write(YAML_mmlu_geo_string)\n",
    "\n",
    "DOC_TO_TEXT = \"\"\"\n",
    "def process_docs(dataset):\n",
    "    def _process_doc(x):\n",
    "        question = x[\"question\"].strip()\n",
    "        choices = x[\"choices\"]\n",
    "        option_a = choices[0]\n",
    "        option_b = choices[1]\n",
    "        option_c = choices[2]\n",
    "        option_d = choices[3]\n",
    "        doc[\"input\"] = f\"{question}\\\\nA. {option_a}\\\\nB. {option_b}\\\\nC. {option_c}\\\\nD. {option_d}\\\\nAnswer:\"\n",
    "        return out_doc\n",
    "\n",
    "    return dataset.map(_process_doc)\n",
    "\"\"\"\n",
    "\n",
    "with open(\"utils_process_docs.py\", \"w\") as f:\n",
    "    f.write(DOC_TO_TEXT)\n",
    "\n",
    "!lm_eval \\\n",
    "    --model hf \\\n",
    "    --model_args pretrained=EleutherAI/pythia-2.8b \\\n",
    "    --include_path ./ \\\n",
    "    --tasks demo_mmlu_high_school_geography_function_prompt_2 \\\n",
    "    --limit 10 \\\n",
    "    --output output/demo_mmlu_high_school_geography_function_prompt_2/ \\\n",
    "    --log_samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We hope that this explainer gives you a sense of what can be done with and how to work with LM-Evaluation-Harnes v0.4.0 ! \n",
    "\n",
    "For more information, check out our documentation pages in the `docs/` folder, and if you have questions, please raise them in GitHub issues, or in #lm-thunderdome or #release-discussion on the EleutherAI discord server."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "zAov81vTbL2K"
   ],
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "46f521b73fd943c081c648fd873ebc0a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "48763b6233374554ae76035c0483066f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "4986a21eb560448fa79f4b25cde48951": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6b2d90209ec14230b3d58a74ac9b83bf": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7c5689bc13684db8a22681f41863dddd": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a1d3a8aa016544a78e8821c8f6199e06": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_f61ed33fad754146bdd2ac9db1ba1c48",
       "IPY_MODEL_bfa0af6aeff344c6845e1080a878e92e",
       "IPY_MODEL_fd1ad9e0367d4004aae853b91c3a7617"
      ],
      "layout": "IPY_MODEL_6b2d90209ec14230b3d58a74ac9b83bf"
     }
    },
    "a73f357065d34d7baf0453ae4a8d75e2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "aed3acd2f2d74003b44079c333a0698e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "bfa0af6aeff344c6845e1080a878e92e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7c5689bc13684db8a22681f41863dddd",
      "max": 5669,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_48763b6233374554ae76035c0483066f",
      "value": 5669
     }
    },
    "f61ed33fad754146bdd2ac9db1ba1c48": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a73f357065d34d7baf0453ae4a8d75e2",
      "placeholder": "​",
      "style": "IPY_MODEL_46f521b73fd943c081c648fd873ebc0a",
      "value": "Downloading builder script: 100%"
     }
    },
    "fd1ad9e0367d4004aae853b91c3a7617": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4986a21eb560448fa79f4b25cde48951",
      "placeholder": "​",
      "style": "IPY_MODEL_aed3acd2f2d74003b44079c333a0698e",
      "value": " 5.67k/5.67k [00:00&lt;00:00, 205kB/s]"
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}