intro.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hz4Pnz5yPNqs"
      },
      "source": [
        "## 🎓 𝗗𝗦𝗣: 𝗗emonstrate–𝗦earch–𝗣redict\n",
        "\n",
        "This notebook illustrates the Demonstrate–Search–Predict (𝗗𝗦𝗣) framework on the multi-hop question answering (QA) task.\n",
        "\n",
        "Below, we'll build **five** small 𝗗𝗦𝗣 programs that implement strategies for this task. The programs range from extremely simple ones (programs 1 and 2) to moderate ones (programs 3 and 4). Program 5 considers a fairly advanced pipeline."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AIabtVS3PNqu"
      },
      "source": [
        "### Installation\n",
        "\n",
        "If you haven't installed **DSP** already, let's do that."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cwF7Cy-7PNqu"
      },
      "outputs": [],
      "source": [
        "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
        "    import google.colab \n",
        "    !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",
        "except: pass\n",
        "\n",
        "!pip install -U dsp-ml"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3IRmg8lPNqv"
      },
      "source": [
        "### Setting Up\n",
        "\n",
        "We'll start by setting up the language model (LM) and retrieval model (RM).\n",
        "\n",
        "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",
        "\n",
        "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",
        "\n",
        "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y_8iQUhvPNqv"
      },
      "outputs": [],
      "source": [
        "%load_ext autoreload\n",
        "%autoreload 2\n",
        "\n",
        "try: import google.colab; root_path = 'dsp'\n",
        "except: root_path = '.'\n",
        "\n",
        "import os\n",
        "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",
        "\n",
        "import dsp\n",
        "\n",
        "openai_key = os.getenv('OPENAI_API_KEY')  # or replace with your API key (optional)\n",
        "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",
        "\n",
        "lm = dsp.GPT3(model='text-davinci-002', api_key=openai_key)\n",
        "rm = dsp.ColBERTv2(url=colbert_server)\n",
        "\n",
        "dsp.settings.configure(lm=lm, rm=rm)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yePgbrmoPNqw"
      },
      "source": [
        "### Task Examples\n",
        "\n",
        "Next, let's look at a few examples of the task. Each example consists of a question and one or more gold answers.\n",
        "\n",
        "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",
        "\n",
        "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xF63-LNIPNqw"
      },
      "outputs": [],
      "source": [
        "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
        "         ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
        "         ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
        "         ('In what year was the star of To Hell and Back born?', ['1925']),\n",
        "         ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
        "         ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",
        "\n",
        "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ys3ZT1oFPNqw"
      },
      "source": [
        "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark, but it'll be instructive to use it for illustration."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lRFZ4OifPNqw"
      },
      "outputs": [],
      "source": [
        "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']),\n",
        "       ('What documentary about the Gilgo Beach Killer debuted on A&E?', ['The Killing Season']),\n",
        "       ('Right Back At It Again contains lyrics co-written by the singer born in what city?', ['Gainesville, Florida', 'Gainesville']),\n",
        "       ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', ['1828']),\n",
        "       ('Which author is English: John Braine or Studs Terkel?', ['John Braine']),\n",
        "       ('Anthony Dirrell is the brother of which super middleweight title holder?', ['Andre Dirrell']),\n",
        "       ('In which city is the sports nutrition business established by Oliver Cookson based ?', ['Cheshire', 'Cheshire, UK']),\n",
        "       ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', ['February 13, 1980']),\n",
        "       ('Kyle Moran was born in the town on what river?', ['Castletown', 'Castletown River']),\n",
        "       (\"What is the name of one branch of Robert D. Braun's speciality?\", ['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']),\n",
        "       (\"Where was the actress who played the niece in the Priest film born?\", ['Surrey', 'Guildford, Surrey']),\n",
        "       ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', ['Portrait of a Marriage']),\n",
        "       ('What year was the father of the Princes in the Tower born?', ['1442'])]\n",
        "\n",
        "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qq2ubHWxPNqx"
      },
      "source": [
        "### Program 1: Vanilla GPT-3.5\n",
        "\n",
        "Let's begin by implementing the simplest program for this task. We'll prompt **GPT-3.5** to answer each question based on its internal parameteric knowledge.\n",
        "\n",
        "We'll start by defining the `Template` that defines how we will communicate with the LM.\n",
        "\n",
        "Specifically, the question–answer template (`qa_template`) will include a question and a short answer for each example."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xn1-ieeHPNqx"
      },
      "outputs": [],
      "source": [
        "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",
        "Answer = dsp.Type(prefix=\"Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",
        "\n",
        "qa_template = dsp.Template(instructions=\"Answer questions with short factoid answers.\", question=Question(), answer=Answer())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kxUoHfFCPNqx"
      },
      "source": [
        "Then, let's define the actual program, `vanilla_LM_QA`. It'll accept a string (`question`) and returns another string (its short `answer`)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sBKPbKHnPNqx"
      },
      "outputs": [],
      "source": [
        "def vanilla_LM_QA(question: str) -> str:\n",
        "    demos = dsp.sample(train, k=7)\n",
        "    example = dsp.Example(question=question, demos=demos)\n",
        "\n",
        "    example, completions = dsp.generate(qa_template)(example, stage='qa')\n",
        "    return completions.answer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j4pj4JUjPNqx"
      },
      "source": [
        "Let's invoke the program on a sample question."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hiyjCsAHPNqy",
        "outputId": "33f86944-e7a6-42bb-8658-88046fd2019b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
            "E. L. Doctorow\n"
          ]
        }
      ],
      "source": [
        "print(dev[0].question)\n",
        "print(vanilla_LM_QA(dev[0].question))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LwVW4vXnPNqy"
      },
      "source": [
        "Let's inspect the last call to the LM to learn more about the behavior of the program."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "62OhWP_zPNqz",
        "outputId": "6f469d76-9aee-4fa7-d25b-161ee8e6f233"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "Answer questions with short factoid answers.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Question: ${the question to be answered}\n",
            "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
            "\n",
            "---\n",
            "\n",
            "Question: Which award did the first book of Gary Zukav receive?\n",
            "Answer: U.S. National Book Award\n",
            "\n",
            "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
            "Answer: Foxcatcher\n",
            "\n",
            "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
            "Answer: Kevin Greutert\n",
            "\n",
            "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
            "Answer: Butch Vig\n",
            "\n",
            "Question: What city was the victim of Joseph Druces working in?\n",
            "Answer: Boston, Massachusetts\n",
            "\n",
            "Question: In what year was the star of To Hell and Back born?\n",
            "Answer: 1925\n",
            "\n",
            "Question: Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
            "Answer:\u001b[32m E. L. Doctorow\u001b[0m\n",
            "\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "lm.inspect_history(n=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C2WP7Uc_PNq0"
      },
      "source": [
        "We can see that the program has organized the examples in a simple prompt and generated an answer for the `dev[0]` question we asked.\n",
        "\n",
        "Let's try on the remaining development examples."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9ISxBoQMPNq0",
        "outputId": "4a1e1208-ca15-457c-8f9b-29aad5360f05"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 13/13 [00:00<00:00, 456.27it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Answered 3 / 13 (23.1%) correctly.\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<style  type=\"text/css\" >\n",
              "    #T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
              "          text-align: left;\n",
              "    }    #T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
              "          text-align: left;\n",
              "    }</style><table id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >question</th>        <th class=\"col_heading level0 col1\" >answer</th>        <th class=\"col_heading level0 col2\" >prediction</th>        <th class=\"col_heading level0 col3\" >correct</th>    </tr></thead><tbody>\n",
              "                <tr>\n",
              "                        <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Long Island Serial Killer</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Melbourne, Australia</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1966</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >July 30, 1953</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Hudson River</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Hong Kong</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Venus</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1457</td>\n",
              "                        <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >❌</td>\n",
              "            </tr>\n",
              "    </tbody></table>"
            ],
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7f9bd91ce1c0>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "23.1"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from dsp.evaluation.utils import evaluate\n",
        "\n",
        "evaluate(vanilla_LM_QA, dev)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_m5D27crPNq0"
      },
      "source": [
        "### Program 2: Retrieve-then-Read w/ GPT-3.5\n",
        "\n",
        "The program above relies entirely on the factual knowledge memorized by GPT-3. As we have seen, it's sometimes correct but it's far from reliable.\n",
        "\n",
        "Let's try to address this. Our first improvement to the vanilla **GPT-3.5** strategy is to include search results in the LM prompt, often called a *retrieve-then-read* approach.\n",
        "\n",
        "In particular, we'll retrieve the most relevant passage to the question (according to the retriever) from the corpus and concatenate it into the prompt. This may help GPT-3.5 answer more factually.\n",
        "\n",
        "To do this, we'll need to modify the `Template`. In particular, we'll define a `qa_template_with_passages` template, which will include a `context` field in addition to a question and an answer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-I8XLvZ5PNq0"
      },
      "outputs": [],
      "source": [
        "Context = dsp.Type(\n",
        "    prefix=\"Context:\\n\",\n",
        "    desc=\"${sources that may contain relevant content}\",\n",
        "    format=dsp.passages2text\n",
        ")\n",
        "\n",
        "qa_template_with_passages = dsp.Template(\n",
        "    instructions=qa_template.instructions,\n",
        "    context=Context(), question=Question(), answer=Answer()\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TKfX2wVVPNq0"
      },
      "source": [
        "Now let's modify the program. We'll call `dsp.retrieve`, which we hooked earlier in this notebook to a **ColBERTv2** retriever serving a Wikipedia (Dec 2018) index."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9yWrtu4ePNq0"
      },
      "outputs": [],
      "source": [
        "def retrieve_then_read_QA(question: str) -> str:\n",
        "    demos = dsp.sample(train, k=7)\n",
        "    passages = dsp.retrieve(question, k=1)\n",
        "    \n",
        "    example = dsp.Example(question=question, context=passages, demos=demos)\n",
        "    example, completions = dsp.generate(qa_template_with_passages)(example, stage='qa')\n",
        "\n",
        "    return completions.answer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zUEBIFuwPNq0"
      },
      "source": [
        "Let's invoke this on a sample question and let's inspect the last call to the LM to learn more about the behavior of the program."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ObXKaGlyPNq0",
        "outputId": "66633c00-7379-4da0-a621-c01ee4536882"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "Answer questions with short factoid answers.\n",
            "\n",
            "---\n",
            "\n",
            "Question: Which award did the first book of Gary Zukav receive?\n",
            "Answer: U.S. National Book Award\n",
            "\n",
            "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
            "Answer: Foxcatcher\n",
            "\n",
            "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
            "Answer: Kevin Greutert\n",
            "\n",
            "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
            "Answer: Butch Vig\n",
            "\n",
            "Question: What city was the victim of Joseph Druces working in?\n",
            "Answer: Boston, Massachusetts\n",
            "\n",
            "Question: In what year was the star of To Hell and Back born?\n",
            "Answer: 1925\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context:\n",
            "${sources that may contain relevant content}\n",
            "\n",
            "Question: ${the question to be answered}\n",
            "\n",
            "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "«Julia Peterkin | Julia Peterkin Julia Peterkin (October 31, 1880 – August 10, 1961) was an American author from South Carolina. In 1929 she won the Pulitzer Prize for Novel/Literature, for her novel \"Scarlet Sister Mary.\" She wrote several novels about the plantation South, especially the Gullah people of the Low Country. She was one of the few white authors who wrote about the African-American experience. Julia Mood was born in Laurens County, South Carolina. Her father was a physician, and she was the third of his four children. Her mother died soon after her birth, and her father later married Janie Brogdon.»\n",
            "\n",
            "Question: Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
            "\n",
            "Answer:\u001b[32m E. L. Doctorow\u001b[0m\n",
            "\n",
            "\n",
            "\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "('E. L. Doctorow', None)"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "retrieve_then_read_QA(dev[0].question), lm.inspect_history(n=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MrrQl9XQPNq0"
      },
      "source": [
        "Now, let's evaluate this on the dev examples."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YJG3MvfWPNq0",
        "outputId": "9bc91b20-4480-430e-f801-f596d7622232"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 13/13 [00:00<00:00, 376.92it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Answered 5 / 13 (38.5%) correctly.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<style  type=\"text/css\" >\n",
              "    #T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
              "          text-align: left;\n",
              "    }    #T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
              "          text-align: left;\n",
              "    }</style><table id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >question</th>        <th class=\"col_heading level0 col1\" >answer</th>        <th class=\"col_heading level0 col2\" >prediction</th>        <th class=\"col_heading level0 col3\" >correct</th>    </tr></thead><tbody>\n",
              "                <tr>\n",
              "                        <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Daytona Beach, Florida.</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1854</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine is English.</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >August 5, 1962</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Dundalk, Ireland</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >Jet Propulsion</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Enid, Oklahoma</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1442</td>\n",
              "                        <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
              "            </tr>\n",
              "    </tbody></table>"
            ],
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7f9bb40bea30>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "38.5"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "evaluate(retrieve_then_read_QA, dev)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_D9AMaS0PNq0"
      },
      "source": [
        "### Program 3: Retrieve-then-Read w/ Self-Consistency\n",
        "\n",
        "It's clear that retrieval has the capacity to help **GPT-3.5** answer a larger number of questions more factually. However, these questions are too complicated for a single retrieved passage to suffice.\n",
        "\n",
        "In this program, we try to make some improvements to the pipeline from **Program 2**.\n",
        "\n",
        "In particular:\n",
        "\n",
        "- We will include 5 passages (instead of a single passage) into the prompt.\n",
        "- We will ask GPT-3.5 to generate a Chain-of-Thought (CoT) rationale to more effectively extract the answer from the passages.\n",
        "- We will use Self-Consistency (SC) to marginalize this prediction across many chains of thought."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KPvVAtSIPNq1"
      },
      "outputs": [],
      "source": [
        "Rationale = dsp.Type(\n",
        "    prefix=\"Rationale: Let's think step by step.\",\n",
        "    desc=\"${a step-by-step deduction that identifies the correct response, which will be provided below}\"\n",
        ")\n",
        "\n",
        "qa_template_with_CoT = dsp.Template(\n",
        "    instructions=qa_template.instructions,\n",
        "    context=Context(), question=Question(), rationale=Rationale(), answer=Answer()\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6gouAgw5PNq1"
      },
      "source": [
        "The snippet above updates our template to include a `rationale` field.\n",
        "\n",
        "Below, we define the new program `retrieve_then_read_QA_v2`. In it, we define a new DSP _transformation_ `QA_predict`, which we will re-use in multiple programs.\n",
        "\n",
        "The transformation `QA_predict` takes an `Example` (which has a question, context, and demonstrations) and generates an answer. It can be configured to use self-consistency (`sc=True`) or not."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JF6eJgulPNq1"
      },
      "outputs": [],
      "source": [
        "@dsp.transformation\n",
        "def QA_predict(example: dsp.Example, sc=True):\n",
        "    if sc:\n",
        "        example, completions = dsp.generate(qa_template_with_CoT, n=20, temperature=0.7)(example, stage='qa')\n",
        "        completions = dsp.majority(completions)\n",
        "    else:\n",
        "        example, completions = dsp.generate(qa_template_with_CoT)(example, stage='qa')\n",
        "    \n",
        "    return example.copy(answer=completions.answer)\n",
        "\n",
        "def retrieve_then_read_QA_v2(question: str) -> str:\n",
        "    demos = dsp.sample(train, k=7)\n",
        "    passages = dsp.retrieve(question, k=5)\n",
        "    example = dsp.Example(question=question, context=passages, demos=demos)\n",
        "    \n",
        "    return QA_predict(example).answer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7gm7nuGPNq1"
      },
      "source": [
        "Let's inspect an example."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Uv1verF8PNq1",
        "outputId": "604be7e1-78fc-4e7f-ccf6-dcc168f5adf9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "Answer questions with short factoid answers.\n",
            "\n",
            "---\n",
            "\n",
            "Question: Which award did the first book of Gary Zukav receive?\n",
            "Answer: U.S. National Book Award\n",
            "\n",
            "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
            "Answer: Foxcatcher\n",
            "\n",
            "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
            "Answer: Kevin Greutert\n",
            "\n",
            "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
            "Answer: Butch Vig\n",
            "\n",
            "Question: What city was the victim of Joseph Druces working in?\n",
            "Answer: Boston, Massachusetts\n",
            "\n",
            "Question: In what year was the star of To Hell and Back born?\n",
            "Answer: 1925\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context:\n",
            "${sources that may contain relevant content}\n",
            "\n",
            "Question: ${the question to be answered}\n",
            "\n",
            "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
            "\n",
            "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
            "[2] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
            "[3] «Right Back at It Again | band played \"Right Back at It Again\" on both occasions. AbsolutePunk writer Thomas Nassiff said the \"mosh call\" lyric \"\"Brace for impact\"\" had \"pretty much ruin[ed] that track\". Tamsyn Wilce for Alter the Press! noted that the way \"Right Back at it Again\" follows on from opening track \"City of Ocala\" \"continues the catchy hardcore beats\" that is present in all of the band's albums. \"Rock Sound\"'s Andy Ritchie called the song \"unmistakably the 'All I Want' of 'Common Courtesy'\". A music video for the song was released on 19 December 2013. The music video was nominated for Best Video»\n",
            "[4] «Born Again in Dixieland | Born Again in Dixieland \"Born Again in Dixieland\" is a song co-written and recorded by Canadian country music artist Jason McCoy. Released in June 1997, it was the lead single from his album \"Playin' for Keeps\". The song reached #3 on the \"RPM\" Country Tracks chart in September 1997. The song won a SOCAN Song of the Year award. McCoy said that he started writing the song with Sheridan and he was playing songs for MCA Canada label and they asked if he had any other songs. He played a verse and the chorus from the song thinking it wasn't»\n",
            "[5] «Right Back Where We Started From | 1974. In a 3 May 2008 interview with Michael Shelley of WFMU, Edwards recalled that after hearing Maxine Nightingale sing on the session for Al Matthews' \"Fool\" that track's producer Pierre Tubbs had come up with \"Right Back Where We Started From\" as a good title for a song for Nightingale herself to record and had invited Edwards to co-write the song. Utilizing a tune which Edwards had written \"a couple of years before\", Tubbs and Edwards wrote \"Right Back Where We Started From\" in about seven minutes while driving to Charing Cross Hospital where Tubbs' wife Gabrielle (née Zimmerman)»\n",
            "\n",
            "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
            "\n",
            "Rationale: Let's think step by step.\u001b[32m The song is by A Day to Remember, and the lyrics were co-written by the singer, Jeremy McKinnon. We just need to find out where McKinnon was born.\n",
            "\n",
            "Answer: Orlando, Florida\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
            "\n",
            "\n",
            "\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "('Ocala, Florida', None)"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "retrieve_then_read_QA_v2(dev[2].question), lm.inspect_history(n=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pt5npUlhPNq1"
      },
      "source": [
        "And let's run this on the other dev examples."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iaoky_dcPNq1",
        "outputId": "866fb8d8-5704-486d-92d2-40796957b22c"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 13/13 [00:01<00:00, 11.23it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Answered 7 / 13 (53.8%) correctly.\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<style  type=\"text/css\" >\n",
              "    #T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
              "          text-align: left;\n",
              "    }    #T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
              "          text-align: left;\n",
              "    }</style><table id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >question</th>        <th class=\"col_heading level0 col1\" >answer</th>        <th class=\"col_heading level0 col2\" >prediction</th>        <th class=\"col_heading level0 col3\" >correct</th>    </tr></thead><tbody>\n",
              "                <tr>\n",
              "                        <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Ocala, Florida</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1824</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >September 27, 1974</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >Aeronautics and Astronautics</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Italy</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1442</td>\n",
              "                        <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
              "            </tr>\n",
              "    </tbody></table>"
            ],
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7f9baab4a850>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "53.8"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "evaluate(retrieve_then_read_QA_v2, dev)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6jkVdeenPNq1"
      },
      "source": [
        "### Program 4: Multi-Hop Retrieval\n",
        "\n",
        "From these simple programs, it becomes clear that a single search query is not enough for this task. This can be seen when trying to find the birth city of the writer of \"Right Back At It Again\". A search query identifies the author correctly as \"Jeremy McKinnon\", but it doesn't mention when he was born.\n",
        "\n",
        "The standard approach for this challenge in the retrieval-augmented NLP literature is to build multi-hop search systems, like GoldEn (Qi et al., 2019), IRRR (Qi et al., 2021) and Baleen (Khattab et al., 2021). These systems read the retrieved results and then generate additional queries to gather additional information if necessary."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_r0dS5c6PNq1"
      },
      "source": [
        "\n",
        "With **DSP**, we can easily simulate such systems in a few lines of code. Let's begin by defining two new templates.\n",
        "\n",
        "First, in `rewrite_template`, we will re-write the input question into a simple search query. This will serve as our \"first hop\" query."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bPDeGaYLPNq1"
      },
      "outputs": [],
      "source": [
        "SearchRationale = dsp.Type(\n",
        "    prefix=\"Rationale: Let's think step by step. To answer this question, we first need to find out\",\n",
        "    desc=\"${the missing information}\"\n",
        ")\n",
        "\n",
        "SearchQuery = dsp.Type(\n",
        "    prefix=\"Search Query:\",\n",
        "    desc=\"${a simple question for seeking the missing information}\"\n",
        ")\n",
        "\n",
        "rewrite_template = dsp.Template(\n",
        "    instructions=\"Write a search query that will help answer a complex question.\",\n",
        "    question=Question(), rationale=SearchRationale(), query=SearchQuery()\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dXsKrFv8PNq1"
      },
      "source": [
        "Next, in `hop_template`, we will use the retrieved information from earlier hops to generate additional search queries for missing information."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D1oiHfIWPNq1"
      },
      "outputs": [],
      "source": [
        "CondenseRationale = dsp.Type(\n",
        "    prefix=\"Rationale: Let's think step by step. Based on the context, we have learned the following.\",\n",
        "    desc=\"${information from the context that provides useful clues}\"\n",
        ")\n",
        "\n",
        "hop_template = dsp.Template(\n",
        "    instructions=rewrite_template.instructions,\n",
        "    context=Context(), question=Question(), rationale=CondenseRationale(), query=SearchQuery()\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y3cdFJKpPNq1"
      },
      "source": [
        "Let's build our `multihop_QA_v1` program. The hallmark of this program is the search transformation called `multihop_search_v1`.\n",
        "\n",
        "This will largely simulate the IRRR system. In each hop, we will retrieve `k` (e.g., `k=2`) passages and concatenate them to the context retrieved from earlier hops to form a `context` chain.\n",
        "\n",
        "For simplicity, we will repeat this for a total of `max_hops` hops (e.g., `max_hops=2`). It's easy to add a stopping criteria in which the LM is asked whether more hops are required."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PxCrlnV9PNq1"
      },
      "outputs": [],
      "source": [
        "from dsp.utils import deduplicate\n",
        "\n",
        "@dsp.transformation\n",
        "def multihop_search_v1(example: dsp.Example, max_hops=2, k=2) -> dsp.Example:\n",
        "    example.context = []\n",
        "    \n",
        "    for hop in range(max_hops):\n",
        "        # Generate a query based\n",
        "        template = rewrite_template if hop == 0 else hop_template\n",
        "        example, completions = dsp.generate(template)(example, stage=f'h{hop}')\n",
        "\n",
        "        # Retrieve k results based on the query generated\n",
        "        passages = dsp.retrieve(completions.query, k=k)\n",
        "\n",
        "        # Update the context by concatenating old and new passages\n",
        "        example.context = deduplicate(example.context + passages)\n",
        "\n",
        "    return example\n",
        "\n",
        "\n",
        "def multihop_QA_v1(question: str) -> str:\n",
        "    demos = dsp.sample(train, k=7)\n",
        "    x = dsp.Example(question=question, demos=demos)\n",
        "    \n",
        "    x = multihop_search_v1(x)\n",
        "    x = QA_predict(x, sc=False)\n",
        "\n",
        "    return x.answer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "idXNnLMDPNq2"
      },
      "source": [
        "Let's run this on one example and inspect the last 3 calls to the LM (i.e., search query 1, search query 2, and QA)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "47rOOBirPNq2",
        "outputId": "a4d87389-efd5-4044-c64e-77a980d1ade7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "Write a search query that will help answer a complex question.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Question: ${the question to be answered}\n",
            "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",
            "Search Query: ${a simple question for seeking the missing information}\n",
            "\n",
            "---\n",
            "\n",
            "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
            "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the singer\n",
            "Search Query: Who sings Right Back At It Again\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Write a search query that will help answer a complex question.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context:\n",
            "${sources that may contain relevant content}\n",
            "\n",
            "Question: ${the question to be answered}\n",
            "\n",
            "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",
            "\n",
            "Search Query: ${a simple question for seeking the missing information}\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
            "[2] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
            "\n",
            "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
            "\n",
            "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m The song is called \"Right Back at It Again\" and it is by the band A Day to Remember. The singer, Jeremy McKinnon, wrote the lyrics.\n",
            "\n",
            "Search Query: Where was Jeremy McKinnon born?\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Answer questions with short factoid answers.\n",
            "\n",
            "---\n",
            "\n",
            "Question: Which award did the first book of Gary Zukav receive?\n",
            "Answer: U.S. National Book Award\n",
            "\n",
            "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
            "Answer: Foxcatcher\n",
            "\n",
            "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
            "Answer: Kevin Greutert\n",
            "\n",
            "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
            "Answer: Butch Vig\n",
            "\n",
            "Question: What city was the victim of Joseph Druces working in?\n",
            "Answer: Boston, Massachusetts\n",
            "\n",
            "Question: In what year was the star of To Hell and Back born?\n",
            "Answer: 1925\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context:\n",
            "${sources that may contain relevant content}\n",
            "\n",
            "Question: ${the question to be answered}\n",
            "\n",
            "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
            "\n",
            "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
            "[2] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
            "[3] «Jeremy McKinnon | Jeremy McKinnon Jeremy Wade McKinnon (born December 17, 1985) is an American musician and record producer, best known as a founding member and the lead vocalist of A Day to Remember. He has produced full-length albums for The Devil Wears Prada, The Ghost Inside, Neck Deep, and Wage War. Jeremy McKinnon was born in Gainesville, Florida and grew up with two sisters. His parents are originally from Brooklyn, New York and is of Irish and Italian descent. McKinnon first got a job at Boston Market and later went on to do construction work. McKinnon's interest in music came as a»\n",
            "[4] «Martin McKinnon | Martin McKinnon Martin \"Marty\" McKinnon (born 5 July 1975 in Adelaide) is a former Australian rules footballer who played with Adelaide, Geelong and the Brisbane Lions in the Australian Football League (AFL). McKinnon was recruited by Adelaide in the 1992 AFL Draft, with their first ever national draft pick. He was the youngest player on Adelaide's list at the time and played for Central District in the SANFL when not appearing with Adelaide. He made his AFL debut late in the 1994 AFL season and was a regular in the team the following year, playing mostly as a wingman. He»\n",
            "\n",
            "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
            "\n",
            "Rationale: Let's think step by step.\u001b[32m The song \"Right Back at It Again\" is co-written by the singer Jeremy McKinnon. A quick search of \"Jeremy McKinnon\" reveals that he was born in Gainesville, Florida.\n",
            "\n",
            "Answer: Gainesville, Florida\u001b[0m\n",
            "\n",
            "\n",
            "\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "('Gainesville, Florida', None)"
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "multihop_QA_v1(dev[2].question), lm.inspect_history(n=3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "37Yyzs0zPNq2"
      },
      "source": [
        "Now, let's run this on the remaining dev examples."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wxmODULzPNq2",
        "outputId": "536f3f86-a5fd-4506-94aa-9dd01f2c623b"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 13/13 [00:00<00:00, 167.43it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Answered 7 / 13 (53.8%) correctly.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<style  type=\"text/css\" >\n",
              "    #T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
              "          text-align: left;\n",
              "    }    #T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
              "          text-align: left;\n",
              "    }</style><table id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >question</th>        <th class=\"col_heading level0 col1\" >answer</th>        <th class=\"col_heading level0 col2\" >prediction</th>        <th class=\"col_heading level0 col3\" >correct</th>    </tr></thead><tbody>\n",
              "                <tr>\n",
              "                        <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Long Island Serial Killer: A Mother's Hunt for Justice</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1947</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >UK</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >September 28, 1969</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >space technology</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1442</td>\n",
              "                        <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
              "            </tr>\n",
              "    </tbody></table>"
            ],
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7f9bb041dc10>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "53.8"
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "evaluate(multihop_QA_v1, dev)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "drWfyQQfPNq2"
      },
      "source": [
        "### Program 5: Multi-Hop Condensed Retrieval w/ Automatic Demos and Query Fusion\n",
        "\n",
        "Through Program 4, we've begun to explore some of the power of the DSP abstraction. However, if you look closely, you will see a few downsides of the previous approach:\n",
        "\n",
        "1. The search transformations invoke the LM without any demonstrations in the prompt. That is because we only have training data for the question–answer pairs and not for the intermediate labels (e.g., search queries).\n",
        "2. The QA prompt uses passages (`context`) and a Chain-of-Thought (`rationale`) for the question to be answered. However, the training demonstrations include neither context nor CoT because they aren't available in our labels.\n",
        "3. The search transformations commit to a single query per hop, which may single out an unproductive chain of passages and hence fail to uncover relevant information."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q-hRtif4PNq2"
      },
      "source": [
        "We address these problems automatically in the program below.\n",
        "\n",
        "In it, we begin by defining `multihop_demonstrate` (which uses `multihop_attempt`) to automatically **annotate** demonstrations for the complex multi-hop pipeline. These demonstrations will be provided to the LM when it's invoked for each transformation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pqNuNwJuPNq2"
      },
      "outputs": [],
      "source": [
        "@dsp.transformation\n",
        "def multihop_attempt(d: dsp.Example) -> dsp.Example:\n",
        "    # Prepare unaugmented demonstrations for the example.\n",
        "    x = dsp.Example(question=d.question, demos=dsp.all_but(train, d))\n",
        "    \n",
        "    # Search. And skip examples where search fails.\n",
        "    # Annotate demonstrations for multihop_search_v2 with the simpler multihop_search_v1 pipeline.\n",
        "    x = multihop_search_v1(x)\n",
        "    if not dsp.passage_match(x.context, d.answer): return None\n",
        "    \n",
        "    # Predict. And skip examples where predict fails.\n",
        "    x = QA_predict(x, sc=False)\n",
        "    if not dsp.answer_match(x.answer, d.answer): return None\n",
        "    \n",
        "    return d.copy(**x)\n",
        "\n",
        "@dsp.transformation\n",
        "def multihop_demonstrate(x: dsp.Example) -> dsp.Example:\n",
        "    demos = dsp.sample(train, k=7)\n",
        "    x.demos = dsp.annotate(multihop_attempt)(demos, k=3, return_all=True)\n",
        "    return x"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zV8ddUJEPNq2"
      },
      "source": [
        "We now implement `multihop_search_v2` as part of `multihop_QA_v2`.\n",
        "\n",
        "In addition to the changes mentioned earlier, this program simulates the Baleen system (Khattab et al., 2021) in a few lines of code.\n",
        "\n",
        "In each retrieval hop (after the very first hop), the summary of the previous hop(s) is included in the prompt. This allows us to efficiently read a larger number of passages from the current hop."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Febt9qzwPNq2"
      },
      "outputs": [],
      "source": [
        "@dsp.transformation\n",
        "def multihop_search_v2(example: dsp.Example, max_hops=2, k=5) -> dsp.Example:\n",
        "    example.context = []\n",
        "\n",
        "    for hop in range(max_hops):\n",
        "        # Generate queries\n",
        "        template = rewrite_template if hop == 0 else hop_template\n",
        "        example, completions = dsp.generate(template, n=10, temperature=0.7)(example, stage=f'h{hop}')\n",
        "        \n",
        "        # Collect the queries and search with result fusion\n",
        "        queries = [c.query for c in completions] + [example.question]\n",
        "        example.context = dsp.retrieveEnsemble(queries, k=k)\n",
        "\n",
        "        # Arrange the passages for the next hop\n",
        "        if hop > 0:\n",
        "            example.context = [completions[0].rationale] + example.context\n",
        "    \n",
        "    return example\n",
        "\n",
        "def multihop_QA_v2(question: str) -> str:\n",
        "    x = dsp.Example(question=question)\n",
        "    x = multihop_demonstrate(x)\n",
        "    x = multihop_search_v2(x)\n",
        "    x = QA_predict(x)\n",
        "    return x.answer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PwRkylaRPNq2",
        "outputId": "3b5cefe1-f5f0-4daa-868e-640820d7e128"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "Write a search query that will help answer a complex question.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Question: ${the question to be answered}\n",
            "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",
            "Search Query: ${a simple question for seeking the missing information}\n",
            "\n",
            "---\n",
            "\n",
            "Question: Which award did the first book of Gary Zukav receive?\n",
            "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the first book of Gary Zukav.\n",
            "Search Query: \"What is the name of Gary Zukav's first book?\"\n",
            "\n",
            "---\n",
            "\n",
            "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
            "Rationale: Let's think step by step. To answer this question, we first need to find out who the heir to the Du Pont family fortune is.\n",
            "Search Query: \"Heir to the Du Pont family fortune\"\n",
            "\n",
            "---\n",
            "\n",
            "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
            "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the 2009 movie featuring Peter Outerbridge as William Easton.\n",
            "Search Query: \"2009 movie featuring Peter Outerbridge as William Easton\"\n",
            "\n",
            "---\n",
            "\n",
            "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
            "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the party of the winner of the 1971 San Francisco mayoral election.\n",
            "Search Query: \"Party of the winner of the 1971 San Francisco mayoral election\"\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Write a search query that will help answer a complex question.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context:\n",
            "${sources that may contain relevant content}\n",
            "\n",
            "Question: ${the question to be answered}\n",
            "\n",
            "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",
            "\n",
            "Search Query: ${a simple question for seeking the missing information}\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
            "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
            "\n",
            "Question: Which award did the first book of Gary Zukav receive?\n",
            "\n",
            "Rationale: Let's think step by step. Based on the context, we have learned the following. Gary Zukav is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. His first book, \"The Dancing Wu Li Masters\" was written with extensive help from Jack Sarfatti and other physicists.\n",
            "\n",
            "Search Query: Which award did \"The Dancing Wu Li Masters\" receive?\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
            "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
            "\n",
            "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
            "\n",
            "Rationale: Let's think step by step. Based on the context, we have learned the following. The Du Pont family is a wealthy American family with a long history of business and philanthropic ventures. In the 1980s, one of the heirs, John Eleuthère du Pont, became interested in wrestling and established a wrestling facility at his farm.\n",
            "\n",
            "Search Query: Which wrestling team did John Eleuthère du Pont sponsor?\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
            "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
            "\n",
            "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
            "\n",
            "Rationale: Let's think step by step. Based on the context, we have learned the following. Outerbridge was born and raised in Toronto, he is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film.\n",
            "\n",
            "Search Query: Who was the director of the 2009 movie \"Saw VI\"?\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «1971 San Francisco mayoral election | 1971 San Francisco mayoral election The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote, from among 11 candidates, there being no provision for a runoff. The three major contenders—Joe Alioto; Harold Dobbs, former member of the Board of Supervisors; and Dianne Feinstein, president of the Board of Supervisors—tried to outdo one another as advocates for law and order. Alioto challenged Dobbs to commit to keeping Police Chief Donald Scott and accused Feinstein of wanting to create a civilian review board to investigate police conduct.»\n",
            "[2] «1971 San Francisco mayoral election | the November 2 election was the re-election of Mayor Joseph Alioto by a 3–2 margin over his nearest challenger. Alioto quoted Jackie Gleason—\"How sweet it is!\"—at a victory celebration and referred to his federal indictment as an attempt by the \"national administration\" to \"knock out a mayor\". Among other races on the ballot was Richard Hongisto's surprise victory for sheriff over incumbent Matthew Carberry. On the ballot was a referendum limiting new-building heights to six stories, which was defeated. A referendum to make the school board an elective one, it being the only appointive board in the state, passed. The»\n",
            "[3] «1971 San Francisco mayoral election | school-board measure was regarded as a reaction to busing initiated by the school board under court order. Another measure approved was to keep cable car service at no less than current levels. 1971 San Francisco mayoral election The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote, from among 11 candidates, there being no provision for a runoff. The three major contenders—Joe Alioto; Harold Dobbs, former member of the Board of Supervisors; and Dianne Feinstein, president of the Board of Supervisors—tried to outdo one another»\n",
            "[4] «1971 San Francisco mayoral election | achieve ethnic balance in elementary schools. Former newspaper editor Scott Newhall proposed staging a world ecology fair and replacing California's water plan by importing icebergs, and imputed little difference between the top three contenders, whom he lumped together as \"Joe Dobbstein\". Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official. As of November, Dobbs, the only Republican in the race, had passed Feinstein in polls and was gaining on Alioto. The result of»\n",
            "[5] «2003 San Francisco mayoral election | against leftist/progressive Gonzalez, a member of the Green Party. It was the first mayoral election in San Francisco that a Green Party candidate took a noticeable amount of the vote. The election was close, with Gonzalez leading in the polls and winning the popular vote among ballots cast on election day, while Newsom had a larger lead on absentee ballots. 2003 San Francisco mayoral election The 2003 San Francisco mayoral election occurred on November 4, 2003. The incumbent, Willie Brown, was termed out of office and could not seek a third term. The general election included three top candidates including»\n",
            "\n",
            "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
            "\n",
            "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote. Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official.\n",
            "\n",
            "Search Query: What year was the Democratic Party founded?\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Answer questions with short factoid answers.\n",
            "\n",
            "---\n",
            "\n",
            "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
            "Answer: Butch Vig\n",
            "\n",
            "Question: What city was the victim of Joseph Druces working in?\n",
            "Answer: Boston, Massachusetts\n",
            "\n",
            "Question: In what year was the star of To Hell and Back born?\n",
            "Answer: 1925\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context:\n",
            "${sources that may contain relevant content}\n",
            "\n",
            "Question: ${the question to be answered}\n",
            "\n",
            "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
            "\n",
            "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
            "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
            "[3] «The Dancing Wu Li Masters | The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980 U.S. National Book Award in category of Science. Although it explores empirical topics in modern physics research, \"The Dancing Wu Li Masters\" gained attention for leveraging metaphors taken from eastern spiritual movements, in particular the Huayen school of Buddhism with the monk Fazang's treatise on The Golden Lion, to explain quantum phenomena and has been regarded by some reviewers as a New Age work, although the»\n",
            "[4] «The Dancing Wu Li Masters | a 1976 physics conference of eastern and western scientists at Esalen Institute, California; and he used the occasion as material for his book. At the conference, it was said that the Chinese term for physics is 'Wu Li', or \"patterns of organic energy.\" Zukav, among others, conceptualized 'physics' as the dance of the Wu Li Masters – teachers of physical essence. Zukav explains the concept further: The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980»\n",
            "\n",
            "Question: Which award did the first book of Gary Zukav receive?\n",
            "\n",
            "Rationale: Let's think step by step. The first book is \"The Dancing Wu Li Masters\" and it was awarded a U.S. National Book Award.\n",
            "\n",
            "Answer: U.S. National Book Award\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
            "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
            "[3] «John du Pont | Foxcatcher team. Du Pont became a sponsor in wrestling, swimming, track, and the modern pentathlon. He was also involved in promoting a subset of the modern pentathlon (run, swim, shoot) as a separate event. He took up athletics and became a competitive wrestler in his 50s. His only prior wrestling experience was as a freshman in high school. He began competing again at the age of 55 in the 1992 Veteran's World Championships in Cali, Colombia; following that in 1993 in Toronto, Ontario; in 1994 in Rome, Italy; and in 1995 in Sofia, Bulgaria. In August 1988, a problem-plagued wrestling»\n",
            "\n",
            "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
            "\n",
            "Rationale: Let's think step by step. The question is asking for a wrestling team that the heir to the Du Pont family sponsored. In the context, it is mentioned that the heir, John du Pont, was interested in wrestling and even competed in wrestling competitions in his 50s. Furthermore, it is stated that he sponsored a wrestling team. Therefore, the answer must be a wrestling team.\n",
            "\n",
            "Answer: Foxcatcher team\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
            "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
            "[3] «Saw VI | Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus on the posthumous effects of the Jigsaw Killer and the progression of his successor, Mark Hoffman. In this film, Hoffman sets a series of traps for an insurance executive, William»\n",
            "[4] «Saw VI | is a crafty move on the screenwriters' part, but once you get past that pseudo-ironic touch, this Saw is no more or less disturbing than any other in the series\". Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus»\n",
            "\n",
            "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
            "\n",
            "Rationale: Let's think step by step. The question is asking for the director of the 2009 movie that features Peter Outerbridge as William Easton. We can use process of elimination to figure this out. In source 1, there is no mention of a 2009 movie. In source 2, the only mention of Peter Outerbridge is in the cast list for a 2004 made-for-TV film. In source 3, we see that Outerbridge was in a 2009 movie, Saw VI, and that it was directed by Kevin Greutert. This must be the correct answer, so we can check source 4 to confirm. In source 4, we see that Kevin Greutert is in fact the director of Saw VI.\n",
            "\n",
            "Answer: Kevin Greutert\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote. Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official.»\n",
            "[2] «Democratic Party (UK, 1998) | Democratic Party (UK, 1998) The Democratic Party was a political party in the United Kingdom, founded in November 1998 by Malvern businessman Geoff Southall, who also funded the party. It was described as \"right wing\" or \"hard right\" in news reports, but claimed to occupy the centre-ground of British politics. It aimed to reduce Britain's involvement with the European Union, opposed the adoption of the Euro, called for direct democracy, and argued for limits on immigration. The slogan was \"The will of the people NOT the party\". In 1999 it had a few hundred members, including previous supporters of James»\n",
            "[3] «Democratic Party (Brazil, 1925-1934) | Democratic Party (Brazil, 1925-1934) The Democratic Party was a Brazilian political party, but state character, founded by dissidents of the Paulista Republican Party (PRP) on February 24, 1925, during the Old Republic and was its manifesto signed by 599 signatories. It was a representative organization of the traditional middle class, linked to coffee sectors, but especially the urban. In May 1926, the Youth Party (Youth Wing of dissenters, who had organized this caption for the federal elections of 1925), also dissident perrepismo, joined the Democratic Party. His most important names were Waldemar Ferreira, José Adriano Marrey Junior, Francisco Morato, Paulo»\n",
            "[4] «Democratic Party (Turkey, 1970) | Democratic Party (Turkey, 1970) Democratic Party (Turkish: \"Demokratik Parti\", DP for short) was a former political party in Turkey founded on 18 December 1970 . It should not be confused with the historical Democrat Party or the current Democrat Party in Turkey. The historical Democrat Party which was the ruling party between 1950-1960 was closed by the military rule. In 1961 and 1965 elections most of ex Democratic Party adherents supported Justice Party, a party founded in 1961. After 1965 elections Justice Party came to power. But following 1969 elections in which Justice Party was again successful, tension between the»\n",
            "[5] «Democratic Party (Japan, 1947) | Democratic Party (Japan, 1947) Democratic Party (民主党, Minshutō) was a centrist political party in Japan. It was founded in spring 1947 by merging the Progressive Party (Shinpo-tō) of Inukai Takeru with a faction of Liberal Party led by Hitoshi Ashida and obtained 124 seats in 1947 elections. The party had held seven seats in Tetsu Katayama's government in 1947-1948. For some months in 1948, party's leader Ashida was Prime minister. In March 1948, part of DP members led by Kijūrō Shidehara joined the Liberal Party to form the Democratic Liberal Party. In 1949 elections, the DP got 69 seats. The»\n",
            "[6] «Democratic Party (United States) | Democratic Party (United States) The Democratic Party is one of the two major contemporary political parties in the United States, along with the Republican Party. Tracing its heritage back to Thomas Jefferson and James Madison's Democratic-Republican Party, the modern-day Democratic Party was founded around 1828 by supporters of Andrew Jackson, making it the world's oldest active political party. The Democrats' dominant worldview was once social conservatism and economic liberalism while populism was its leading characteristic in the rural South. In 1912, Theodore Roosevelt ran as a third-party candidate in the Progressive (\"Bull Moose\") Party, beginning a switch of political platforms»\n",
            "\n",
            "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
            "\n",
            "Rationale: Let's think step by step.\u001b[32m The question is asking for the year the party of the mayor of San Francisco in 1971 was founded. In source 1, we see that the election was in 1971 and the winner was Joseph Alioto. In source 2, we see that the party was founded in 1998, which is not correct. In source 3, we see that the party was founded in 1925, which is not correct. In source 4, we see that the party was founded in 1970, which is not correct. In source 5, we see that the party was founded in 1947, which is not correct. In source 6, we see that the party was founded around 1828. This must be the correct answer, so we can check source 1 to confirm. In source 1\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
            "\n",
            "\n",
            "\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "('1828', None)"
            ]
          },
          "execution_count": 25,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "multihop_QA_v2(dev[3].question), lm.inspect_history(n=3)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "puDzXhVNPNq2",
        "outputId": "d8835ab4-b8a7-4ea1-82cb-42b659a69804"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 13/13 [00:02<00:00,  5.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Answered 11 / 13 (84.6%) correctly.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<style  type=\"text/css\" >\n",
              "    #T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
              "          text-align: left;\n",
              "    }    #T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
              "          text-align: left;\n",
              "    }</style><table id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >question</th>        <th class=\"col_heading level0 col1\" >answer</th>        <th class=\"col_heading level0 col2\" >prediction</th>        <th class=\"col_heading level0 col3\" >correct</th>    </tr></thead><tbody>\n",
              "                <tr>\n",
              "                        <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1828</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >February 13, 1980</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
              "            </tr>\n",
              "            <tr>\n",
              "                        <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1442</td>\n",
              "                        <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
              "            </tr>\n",
              "    </tbody></table>"
            ],
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7f9ba36cfb50>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "84.6"
            ]
          },
          "execution_count": 26,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "evaluate(multihop_QA_v2, dev)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AVpexTEHPNq2"
      },
      "source": [
        "### Ask your own questions\n",
        "\n",
        "You could also ask your own questions. This can be a productive vehicle to find problems in the programs above and implement patches for them! (Keep in mind that the search index we're using currently is from Dec 2018.)\n",
        "\n",
        "To do this, make sure you set your API key. This can be done by defining the `OPENAI_API_KEY` environemt variable or setting it at the beginning of the program (Cell #2)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YuuIbKGUPNq2",
        "outputId": "7cc07ca1-3b79-4342-e15e-fa807a2262d3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "Write a search query that will help answer a complex question.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Question: ${the question to be answered}\n",
            "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",
            "Search Query: ${a simple question for seeking the missing information}\n",
            "\n",
            "---\n",
            "\n",
            "Question: Which award did the first book of Gary Zukav receive?\n",
            "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the first book of Gary Zukav.\n",
            "Search Query: \"What is the name of Gary Zukav's first book?\"\n",
            "\n",
            "---\n",
            "\n",
            "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
            "Rationale: Let's think step by step. To answer this question, we first need to find out who the heir to the Du Pont family fortune is.\n",
            "Search Query: \"Heir to the Du Pont family fortune\"\n",
            "\n",
            "---\n",
            "\n",
            "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
            "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the 2009 movie featuring Peter Outerbridge as William Easton.\n",
            "Search Query: \"2009 movie featuring Peter Outerbridge as William Easton\"\n",
            "\n",
            "---\n",
            "\n",
            "Question: When was the creator of Hadoop given an award?\n",
            "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the creator of Hadoop.\n",
            "Search Query: \"Name of the creator of Hadoop\"\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Write a search query that will help answer a complex question.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context:\n",
            "${sources that may contain relevant content}\n",
            "\n",
            "Question: ${the question to be answered}\n",
            "\n",
            "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",
            "\n",
            "Search Query: ${a simple question for seeking the missing information}\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
            "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
            "\n",
            "Question: Which award did the first book of Gary Zukav receive?\n",
            "\n",
            "Rationale: Let's think step by step. Based on the context, we have learned the following. Gary Zukav is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. His first book, \"The Dancing Wu Li Masters\" was written with extensive help from Jack Sarfatti and other physicists.\n",
            "\n",
            "Search Query: Which award did \"The Dancing Wu Li Masters\" receive?\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
            "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
            "\n",
            "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
            "\n",
            "Rationale: Let's think step by step. Based on the context, we have learned the following. The Du Pont family is a wealthy American family with a long history of business and philanthropic ventures. In the 1980s, one of the heirs, John Eleuthère du Pont, became interested in wrestling and established a wrestling facility at his farm.\n",
            "\n",
            "Search Query: Which wrestling team did John Eleuthère du Pont sponsor?\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
            "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
            "\n",
            "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
            "\n",
            "Rationale: Let's think step by step. Based on the context, we have learned the following. Outerbridge was born and raised in Toronto, he is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film.\n",
            "\n",
            "Search Query: Who was the director of the 2009 movie \"Saw VI\"?\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Apache Hadoop | the Java programming language, with some native code in C and command line utilities written as shell scripts. Though MapReduce Java code is common, any programming language can be used with Hadoop Streaming to implement the map and reduce parts of the user's program. Other projects in the Hadoop ecosystem expose richer user interfaces. According to its co-founders, Doug Cutting and Mike Cafarella, the genesis of Hadoop was the Google File System paper that was published in October 2003. This paper spawned another one from Google \"MapReduce: Simplified Data Processing on Large Clusters\". Development started on the Apache Nutch project,»\n",
            "[2] «Doug Cutting | Doug Cutting Douglass Read Cutting is a software designer and advocate and creator of open-source search technology. He founded Lucene and, with Mike Cafarella, Nutch, both open-source search technology projects which are now managed through the Apache Software Foundation. Cutting and Cafarella are also the co-founders of Apache Hadoop. Cutting holds a bachelor's degree from Stanford University. Prior to developing Lucene, Cutting held search technology positions at Xerox PARC where he worked on the Scatter/Gather algorithm and on computational stylistics. He also worked at Excite, where he was one of the chief designers of the search engine, and Apple Inc.,»\n",
            "[3] «Doug Cutting | September 2010, he was elected its chairman. In 2015, O'Reilly awarded an open source award to Cutting. Doug Cutting Douglass Read Cutting is a software designer and advocate and creator of open-source search technology. He founded Lucene and, with Mike Cafarella, Nutch, both open-source search technology projects which are now managed through the Apache Software Foundation. Cutting and Cafarella are also the co-founders of Apache Hadoop. Cutting holds a bachelor's degree from Stanford University. Prior to developing Lucene, Cutting held search technology positions at Xerox PARC where he worked on the Scatter/Gather algorithm and on computational stylistics. He also worked»\n",
            "[4] «Apache Hadoop | but was moved to the new Hadoop subproject in January 2006. Doug Cutting, who was working at Yahoo! at the time, named it after his son's toy elephant. The initial code that was factored out of Nutch consisted of about 5,000 lines of code for HDFS and about 6,000 lines of code for MapReduce. The first committer to add to the Hadoop project was Owen O'Malley (in March 2006); Hadoop 0.1.0 was released in April 2006. It continues to evolve through contributions that are being made to the project. Hadoop consists of the \"Hadoop Common\" package, which provides file system»\n",
            "[5] «Doug Cutting | on the MapReduce algorithm, which allows very large scale computations to be trivially parallelized across large clusters of servers. Cutting and Cafarella, realizing the importance of this paper to extending Lucene into the realm of extremely large search problems, created the open-source Hadoop framework that allows applications based on the MapReduce paradigm to be run on large clusters of commodity hardware. Cutting was an employee of Yahoo!, where he led the Hadoop project full-time. He later went on to work for Cloudera. In July 2009, Cutting was elected to the board of directors of the Apache Software Foundation, and in»\n",
            "\n",
            "Question: When was the creator of Hadoop given an award?\n",
            "\n",
            "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m Apache Hadoop is an open-source software framework for distributed storage and distributed processing of big data sets on computer clusters. The creator of Hadoop is Doug Cutting. In 2015, O'Reilly awarded an open source award to Cutting.\n",
            "\n",
            "Search Query: When was the creator of Hadoop given an award?\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Answer questions with short factoid answers.\n",
            "\n",
            "---\n",
            "\n",
            "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
            "Answer: Butch Vig\n",
            "\n",
            "Question: What city was the victim of Joseph Druces working in?\n",
            "Answer: Boston, Massachusetts\n",
            "\n",
            "Question: In what year was the star of To Hell and Back born?\n",
            "Answer: 1925\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context:\n",
            "${sources that may contain relevant content}\n",
            "\n",
            "Question: ${the question to be answered}\n",
            "\n",
            "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
            "\n",
            "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
            "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
            "[3] «The Dancing Wu Li Masters | The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980 U.S. National Book Award in category of Science. Although it explores empirical topics in modern physics research, \"The Dancing Wu Li Masters\" gained attention for leveraging metaphors taken from eastern spiritual movements, in particular the Huayen school of Buddhism with the monk Fazang's treatise on The Golden Lion, to explain quantum phenomena and has been regarded by some reviewers as a New Age work, although the»\n",
            "[4] «The Dancing Wu Li Masters | a 1976 physics conference of eastern and western scientists at Esalen Institute, California; and he used the occasion as material for his book. At the conference, it was said that the Chinese term for physics is 'Wu Li', or \"patterns of organic energy.\" Zukav, among others, conceptualized 'physics' as the dance of the Wu Li Masters – teachers of physical essence. Zukav explains the concept further: The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980»\n",
            "\n",
            "Question: Which award did the first book of Gary Zukav receive?\n",
            "\n",
            "Rationale: Let's think step by step. The first book is \"The Dancing Wu Li Masters\" and it was awarded a U.S. National Book Award.\n",
            "\n",
            "Answer: U.S. National Book Award\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
            "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
            "[3] «John du Pont | Foxcatcher team. Du Pont became a sponsor in wrestling, swimming, track, and the modern pentathlon. He was also involved in promoting a subset of the modern pentathlon (run, swim, shoot) as a separate event. He took up athletics and became a competitive wrestler in his 50s. His only prior wrestling experience was as a freshman in high school. He began competing again at the age of 55 in the 1992 Veteran's World Championships in Cali, Colombia; following that in 1993 in Toronto, Ontario; in 1994 in Rome, Italy; and in 1995 in Sofia, Bulgaria. In August 1988, a problem-plagued wrestling»\n",
            "\n",
            "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
            "\n",
            "Rationale: Let's think step by step. The question is asking for a wrestling team that the heir to the Du Pont family sponsored. In the context, it is mentioned that the heir, John du Pont, was interested in wrestling and even competed in wrestling competitions in his 50s. Furthermore, it is stated that he sponsored a wrestling team. Therefore, the answer must be a wrestling team.\n",
            "\n",
            "Answer: Foxcatcher team\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
            "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
            "[3] «Saw VI | Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus on the posthumous effects of the Jigsaw Killer and the progression of his successor, Mark Hoffman. In this film, Hoffman sets a series of traps for an insurance executive, William»\n",
            "[4] «Saw VI | is a crafty move on the screenwriters' part, but once you get past that pseudo-ironic touch, this Saw is no more or less disturbing than any other in the series\". Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus»\n",
            "\n",
            "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
            "\n",
            "Rationale: Let's think step by step. The question is asking for the director of the 2009 movie that features Peter Outerbridge as William Easton. We can use process of elimination to figure this out. In source 1, there is no mention of a 2009 movie. In source 2, the only mention of Peter Outerbridge is in the cast list for a 2004 made-for-TV film. In source 3, we see that Outerbridge was in a 2009 movie, Saw VI, and that it was directed by Kevin Greutert. This must be the correct answer, so we can check source 4 to confirm. In source 4, we see that Kevin Greutert is in fact the director of Saw VI.\n",
            "\n",
            "Answer: Kevin Greutert\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Apache Hadoop is an open-source software framework for distributed storage and distributed processing of big data sets on computer clusters. The creator of Hadoop is Doug Cutting. In 2015, O'Reilly awarded an open source award to Cutting.»\n",
            "[2] «Doug Cutting | September 2010, he was elected its chairman. In 2015, O'Reilly awarded an open source award to Cutting. Doug Cutting Douglass Read Cutting is a software designer and advocate and creator of open-source search technology. He founded Lucene and, with Mike Cafarella, Nutch, both open-source search technology projects which are now managed through the Apache Software Foundation. Cutting and Cafarella are also the co-founders of Apache Hadoop. Cutting holds a bachelor's degree from Stanford University. Prior to developing Lucene, Cutting held search technology positions at Xerox PARC where he worked on the Scatter/Gather algorithm and on computational stylistics. He also worked»\n",
            "[3] «Anoop Swarup | 1984. He initiated the UNDP-NYKS Poverty Alleviation Awards in 2000, was instrumental in launching a ‘National Think Tank’ in the Ministry of Finance in 2004 and campaign on ‘Indian Civilization through the Millennium’ (ICTM) in 2008 now an ongoing yearly event at Hastinapur. He founded the ‘Sambhav’ Trust in the Indian State of Rajasthan for de addiction in 2008. Anoop Swarup is associated with various organizations that include Board of Directors of OISCA Organization for Industrial, Spiritual and Cultural Advancement International, Tokyo Japan, United Nations Inter Agency working group on Population and Development as part of United Nations Development Assistance»\n",
            "[4] «Apache Hadoop | but was moved to the new Hadoop subproject in January 2006. Doug Cutting, who was working at Yahoo! at the time, named it after his son's toy elephant. The initial code that was factored out of Nutch consisted of about 5,000 lines of code for HDFS and about 6,000 lines of code for MapReduce. The first committer to add to the Hadoop project was Owen O'Malley (in March 2006); Hadoop 0.1.0 was released in April 2006. It continues to evolve through contributions that are being made to the project. Hadoop consists of the \"Hadoop Common\" package, which provides file system»\n",
            "[5] «Apache SystemML | November 2, 2015. On May 17, 2017, the Apache Software Foundation Board approved the graduation of Apache SystemML as an Apache Top Level Project. Apache SystemML Apache SystemML is a flexible machine learning system that automatically scales to Spark and Hadoop clusters. SystemML's distinguishing characteristics are: SystemML was created in 2010 by researchers at the IBM Almaden Research Center led by IBM Fellow Shivakumar Vaithyanathan. It was observed that data scientists would write machine learning algorithms in languages such as R and Python for small data. When it came time to scale to big data, a systems programmer would be»\n",
            "[6] «Matei Zaharia | the Apache Mesos project. He also designed and implemented the core scheduling algorithms used in Apache Hadoop. He received two Best Paper awards at NSDI 2012 and SIGCOMM 2012, Honorable Mention for Community Award at NSDI 2012, and a Best Demo Award at SIGMOD 2012. Jointly with Reynold Xin, Parviz Deyhim, Xiangrui Meng, and Ali Ghodsi, he holds the 2014 world record in Daytona GraySort using Apache Spark. In 2015 he received the ACM Doctoral Dissertation Award. Matei Zaharia Matei Zaharia is a Romanian-Canadian computer scientist specializing in big data, distributed systems, and cloud computing. He is a co-founder and»\n",
            "\n",
            "Question: When was the creator of Hadoop given an award?\n",
            "\n",
            "Rationale: Let's think step by step.\u001b[32m The question is asking for the year when the creator of Hadoop was given an award. In source 1, we see that the creator of Hadoop is Doug Cutting and that in 2015, O'Reilly awarded an open source award to Cutting. Therefore, we can infer that the answer is 2015. We can check sources 2 and 3 to see if there is any mention of an award given to Doug Cutting in 2015, but since there is no mention, we can assume that source 1 is correct and the answer is 2015.\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
            "\n",
            "\n",
            "\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "('2015', None)"
            ]
          },
          "execution_count": 27,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "multihop_QA_v2(\"When was the creator of Hadoop given an award?\"), lm.inspect_history(n=3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l3OaH8RAPNq3"
      },
      "source": [
        "### Additional Programs...\n",
        "\n",
        "While Program 5 is undeniably better than, say, Program 1, it's far from perfect.\n",
        "\n",
        "Many different ideas can easily be implemented to make it more general or more powerful. Refer to the DSP paper for inspiration!\n",
        "\n",
        "One of your first improvements to Programs 4 and 5 would be to allow it to do more than two hops and to automatically stop when additional hops are not required.\n",
        "\n",
        "**Hint:** This is pretty easy to do. You can start with Program 4 and modify `hop_template` so its `SearchQuery` produces the string `N/A` when additional hops are not required.\n",
        "\n",
        "In the `multihop_search_v1` function, you can then break from the loop — i.e., `if completions.query == 'N/A': break` whenever this happens. How will you apply this to Program 5?\n",
        "\n",
        "```\n",
        "SearchQueryWithStopping = dsp.Type(\n",
        "    prefix=\"Search Query:\",\n",
        "    desc=\"${a simple question for seeking the missing or required information --- say N/A if the context above contains all of the required information}\"\n",
        ")\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LIgU-SQCPNq3"
      },
      "source": [
        "Other incremental improvements will likely help program 5, including keeping the top passage from each hop in the context and/or the summary of each hop. To decide which changes to make, you can inspect the trace of the program when it makes mistakes!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bX2mWmbqPNq3"
      },
      "outputs": [],
      "source": [
        "# Program 4 with max_hops=4 and automatic stopping."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "base",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "orig_nbformat": 4,
    "vscode": {
      "interpreter": {
        "hash": "de72ef8826336f005e74d4a3f2b2dfb9239c3f097fd15aff69f3f6a4fc3bd8b8"
      }
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}