2_paris_prompts.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {},
      "outputs": [],
      "source": [
        "# MIT License\n",
        "#\n",
        "#@title Copyright (c) 2023 CCAI Community Authors { display-mode: \"form\" }\n",
        "#\n",
        "# Permission is hereby granted, free of charge, to any person obtaining a\n",
        "# copy of this software and associated documentation files (the \"Software\"),\n",
        "# to deal in the Software without restriction, including without limitation\n",
        "# the rights to use, copy, modify, merge, publish, distribute, sublicense,\n",
        "# and/or sell copies of the Software, and to permit persons to whom the\n",
        "# Software is furnished to do so, subject to the following conditions:\n",
        "#\n",
        "# The above copyright notice and this permission notice shall be included in\n",
        "# all copies or substantial portions of the Software.\n",
        "#\n",
        "# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL\n",
        "# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING\n",
        "# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER\n",
        "# DEALINGS IN THE SOFTWARE."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# NLP Models for Climate Policy Analysis Part 2: Prompts for Sustainable Development Goals\n",
        "\n",
        "\n",
        "Author(s):\n",
        "*   Daniel Spokoyny, Carnegie Mellon University, dspokoyn@cs.cmu.edu\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Table of Contents\n",
        "\n",
        "*   [Target Audience](#target-audience)\n",
        "*   [Overview](#overview)\n",
        "*   [Contributions](#contributions)\n",
        "*   [Software Requirements](#software-requirements)\n",
        "*   [Data Source](#data-source)\n",
        "*   [Simple Prompt](#simple-prompt)\n",
        "*   [In-Context Learning Prompts](#icl-prompt)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a name=\"target-audience\"></a>\n",
        "# Target Audience\n",
        "\n",
        "* Climate scientists seeking to use large language models to tackle a domain-specific problem in climate change. \n",
        "* Data scientists with prior background in deep learning who are interested in building prompt-based large language model classifiers for climate texts."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a name=\"Motivation\"></a>\n",
        "# Motivation\n",
        "\n",
        "The Paris Agreement is an international treaty on climate change that was adopted in 2015. \n",
        "One of the results of the treaty is that all participating countries are required to submit a Nationally Determined Contribution (NDC) report every five years.\n",
        "These reports include a variety of vital information, such as the country's current greenhouse gas emissions, their goals for reducing emissions, and the policies they plan to implement to achieve these goals.\n",
        "Since these reports are written in plain text and can be hundreds of pages long, many questions are difficult to answer automatically, such as:\n",
        "*   What are the most common policies that countries are implementing to reduce emissions?\n",
        "*   Which countries are on track to meet their goals?\n",
        "*   Which countries are falling behind?\n",
        "*   What are the most common barriers to reducing emissions?\n",
        "\n",
        "One way to better understand these reports would be to use a common framework to help organize the information.\n",
        "The Sustainable Development Goals (SDG) framework aims to provide a set of 17 climate goals (e.g. \"Affordable and Clean Energy\") with even more subgoals (or targets) that can be used to organize the information in the NDC reports.\n",
        "\n",
        "Climate Watch, is a project by the World Resources Institute that has manually aligned the unstructured texts of the Nationally Determined Contribution reports to the semi-structured Sustainable Development Goals (SDG) framework.\\footnote{https://www.climatewatchdata.org/}\n",
        "By doing so, Climate Watch allows for different stakeholders to more easily compare progress across countries and identify which areas need financing, education, policy changes or other resources.\n",
        "\n",
        "In this tutorial, we will build a prompt-based classifier and use this dataset to evaluate the performance of the classifier.\n",
        "Since, new documents are being updated, a tool that can automatically align NDC reports to the SDG framework would save a lot of time and effort."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a name=\"overview\"></a>\n",
        "# Overview: OpenAI Classification Extraction with Prompt Based In-Context Learning\n",
        "\n",
        "In this tutorial we will use the state of the art Large Language Models (LLMs) to classify climate policy documents.\n",
        "For the data we will work with the Nationally Determined Contributions (NDCs) documents from the Paris Climate Agreement. As the model we will use OpenAI's 'text-davinci-003' and attempt to use carefully crafted prompts to classify sentences from our dataset.\n",
        "\n",
        "While in the previous tutorial we were focused on training supervised NLP models for text classification, with the advent of LLMs a more recent paradigm, prompting, has emerged. Prompting allows for interfacing with a trained machine learning model using text as input and text as output.\n",
        "\n",
        "The rest of the tutorial is organized as follows\n",
        "\n",
        "(1) **Dataset**: We are going to load the dataset and get it ready for prompt classification. \\\n",
        "(2) **Prompting**: We are going to construct a simple prompt and use it to classify sentences. \\\n",
        "(3) **ICL**: We are going to add to the prompt in-context learning examples and also increase the context of the input text. \n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a name=\"contributions\"></a>\n",
        "# Contributions\n",
        "\n",
        "Through introductory presentations and colab notebooks, we aim to expose the participants to: \n",
        "\n",
        "(a) **Dataset**: Outlining a dataset of national climate action plans with corresponding annotations to the Sustainable Development Goals framwork. \\\n",
        "(b) **Prompting**: Designing prompts with in-context learning to perform novel text classification tasks. \\\n",
        "(c) **LLMs** Using large language models (LLMs) via python web-based APIs. \\"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a name=\"software-requirements\"></a>\n",
        "# Software Requirements\n",
        "This notebook requires the following libraries:\n",
        "*   datasets (pip)\n",
        "*   openai (pip)\n",
        "\n",
        "`datasets` is a library which we will use to download the data required.\n",
        "`openai` is a library from which we can access large language models via a python API. To use the model, you will need to sign up for an OpenAI Account (http://platform.openai.com/signup) and get an API key pasted below.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "mspefyudAIiK"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "UsageError: Line magic function `%%capture` not found.\n"
          ]
        }
      ],
      "source": [
        "# We need to install the following packages to run the code in this notebook\n",
        "%%capture\n",
        "!pip install datasets\n",
        "!pip install openai"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/home/dspokoyn/.cache/pypoetry/virtualenvs/tutorial-IHiCU4EY-py3.8/lib/python3.8/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
            "  from .autonotebook import tqdm as notebook_tqdm\n"
          ]
        }
      ],
      "source": [
        "# Import libraries\n",
        "import openai\n",
        "import json\n",
        "from huggingface_hub import hf_hub_download\n",
        "from datasets import load_dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 101,
          "referenced_widgets": [
            "ab879f6b5c0e48e6a88fcef30c4a0dc5",
            "b60848d12d9d4743b648ebaf43122d87",
            "bdf310baec9e4bb292a616a1c9941b44",
            "cc76fa30bf2745899ebb1a4fdc108139",
            "af3ccb7b556948449f317600d1361158",
            "c866b6073eb54001aeede1a87f4c79a2",
            "5b515b9cf3be4941b350a11e24aa8c26",
            "daad8401c7b54fb98b488587d455d7e4",
            "e267c4404b5145fb9465d5a89e86da1b",
            "30058e6b100a4837b21bee31c619f7c7",
            "0a8d3c8e28964a2b83dbf16902bf35a3"
          ]
        },
        "id": "QmHtjYrnAIiL",
        "outputId": "4b197b33-8e37-4ccf-cb8f-da1817fb49df"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Found cached dataset parquet (/home/dspokoyn/.cache/huggingface/datasets/dspoka___parquet/dspoka--sdg-single-c49077ade9802265/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)\n",
            "100%|██████████| 1/1 [00:00<00:00, 195.13it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The dataset has 14219 rows and 13 columns\n",
            "The columns are: ['iso3' 'country' 'goal' 'target' 'text' 'status' 'sector' 'response'\n",
            " 'infotype' 'start' 'end' 'filename' '__index_level_0__']\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        }
      ],
      "source": [
        "# This here is how we will load the SDG dataset from the HuggingFace Hub.\n",
        "# It has already been preprocessed for you to use.\n",
        "dataset = load_dataset(\"dspoka/sdg-single\")\n",
        "\n",
        "# Here we will transform the dataset into a pandas dataframe\n",
        "df_full = dataset['full'].to_pandas()\n",
        "\n",
        "print(f'The dataset has {df_full.shape[0]} rows and {df_full.shape[1]} columns')\n",
        "print(f'The columns are: {df_full.columns.values}')"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a name=\"data-source\"></a>\n",
        "# Understanding the SDG Dataset\n",
        "- The important columns for us to know are 'goal', 'target', and 'text'.\n",
        "- The 'text' column is the raw text that was annotated in the NDC documents. We will feed this into our model and use it to predict the the label.\n",
        "- The 'goal' column is the SDG goal. We will use this as our main label for prediction.\n",
        "- Each SDG 'goal' also has sub-goals or 'target' as the name of the column.\n",
        "- Currently if you take a look the goals they are [\"1\",...,\"17\"]. Goal \"12\" make have a target \"12.2\".\n",
        "- What exactly does goal \"12\" stand for exactly? Well for this we need to take a look at the metadata.\n",
        "- Below, is an image that shows the 2030 progerss for the SDG goals and targets.\n",
        "![SDG Goals](https://unece.org/sites/default/files/styles/max_1300x1300/public/2021-03/Graphic%20EN%20_PR_SDG%20Report.png?itok=Mwuuj3c7)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "IWBRe-PSAIiM"
      },
      "outputs": [],
      "source": [
        "def unpack_and_explain_metadata(metadata):\n",
        "    '''Args:\n",
        "        metadata: dictionary containing the metadata for the dataset.\n",
        "    Returns:\n",
        "        goal_idx_2_description: dictionary mapping goal index to the text description of the goal.\n",
        "        goal_idx_2_goal_number: dictionary mapping goal index to the goal number. The goal number is what is currently in the dataset frame.\n",
        "        target_idx_2_description: dictionary mapping target index to the text description of the target.\n",
        "        target_idx_2_goal_idx: dictionary that maps a target index to the goal index. There are multiple targets per goal.\n",
        "        target_idx_2_target_number: dictionary that maps a target index to the target number. The target number is what is currently in the dataset frame.\n",
        "\n",
        "    '''\n",
        "    goal_idx_2_description = metadata['goal_idx_2_description']\n",
        "    goal_idx_2_description = {int(k):v for k,v in goal_idx_2_description.items()}\n",
        "    \n",
        "    goal_idx_2_goal_number = metadata['goal_idx_2_goal_number']\n",
        "    goal_idx_2_goal_number = {int(k):v for k,v in goal_idx_2_goal_number.items()}\n",
        "    \n",
        "    target_idx_2_description = metadata['target_idx_2_description']\n",
        "    target_idx_2_description = {int(k):v for k,v in target_idx_2_description.items()}\n",
        "    \n",
        "    target_idx_2_goal_idx = metadata['target_idx_2_goal_idx']\n",
        "    target_idx_2_goal_idx = {int(k):int(v) for k,v in target_idx_2_goal_idx.items()}\n",
        "\n",
        "    target_idx_2_target_number = metadata['target_idx_2_target_number']\n",
        "    target_idx_2_target_number = {int(k):v for k,v in target_idx_2_target_number.items()}\n",
        "\n",
        "    return goal_idx_2_description, goal_idx_2_goal_number, target_idx_2_description, target_idx_2_goal_idx, target_idx_2_target_number\n",
        "\n",
        "\n",
        "# We will also download the metadata for the dataset from the HuggingFace Hub.\n",
        "filepath = hf_hub_download(repo_id=\"dspoka/sdg-single\", filename=\"metadata.json\", repo_type=\"dataset\")\n",
        "with open(filepath) as f:\n",
        "    metadata = json.load(f)\n",
        "\n",
        "# In this method we will unpack the dictionaries and explain what each of the metadata means\n",
        "goal_idx_2_description, goal_idx_2_goal_number, target_idx_2_description, target_idx_2_goal_idx, target_idx_2_target_number = unpack_and_explain_metadata(metadata)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### So lets look at a row of our dataset df_full and see how to use the metadata to understand it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "iso3                                                               AFG\n",
            "country                                                    Afghanistan\n",
            "goal                                                                12\n",
            "target                                                            12.2\n",
            "text                 Reduce rural peoples’ dependence on fuel for c...\n",
            "status                                                          future\n",
            "sector                                               energy efficiency\n",
            "response                                                    mitigation\n",
            "infotype                                                needs_and_gaps\n",
            "start                                                          28031.0\n",
            "end                                                            28096.0\n",
            "filename                                         AFG-first_ndc-EN.html\n",
            "__index_level_0__                                                    0\n",
            "Name: 0, dtype: object\n",
            "Text Reduce rural peoples’ dependence on fuel for cooking and heating.\n"
          ]
        }
      ],
      "source": [
        "# first row\n",
        "print(df_full.iloc[0])\n",
        "# and here is the text of the first row:\n",
        "print('Text', df_full.iloc[0]['text'])\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Looking at row\n",
        "- We see that this piece of text is from the NDC of the country \"Afghanistan\" and from this particular file: AFG-first_ndc-EN.html\n",
        "- We also see some other topic labels, such that this is about the \"energy efficiency\" sector, is a type of mitigation Climate Response, and that its current implementation status is \"future\".\n",
        "- We also see that the Sustainable Development Goals (SDG) goal is \"12\" and target is \"12.2\".\n",
        "- Note idx or the index is 0-indexed for both goal_idx and target_idx.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are 17 goals in the dataset.\n",
            "The \"12\" goal has this text description:  Ensure sustainable consumption and production patterns\n"
          ]
        }
      ],
      "source": [
        "goal_number = \"12\"\n",
        "goal_idx = None\n",
        "for k,v in goal_idx_2_goal_number.items():\n",
        "    if v == \"12\":\n",
        "        goal_idx = k\n",
        "        break\n",
        "\n",
        "print(f'There are {len(goal_idx_2_description)} goals in the dataset.')\n",
        "print(f'The \"{goal_number}\" goal has this text description: {goal_idx_2_description[goal_idx]}')"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can also find all targets or subgoals that belong to this goal in specific."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Target 0 has the following description: By 2030, achieve the sustainable management and efficient use of natural resources\n",
            "Target 9 has the following description: By 2030, substantially reduce waste generation through prevention, reduction, recycling and reuse\n",
            "Target 20 has the following description: Implement the 10-year framework of programmes on sustainable consumption and production, all countries taking action, with developed countries taking the lead, taking into account the development and capabilities of developing countries\n",
            "Target 21 has the following description: By 2020, achieve the environmentally sound management of chemicals and all wastes throughout their life cycle, in accordance with agreed international frameworks, and significantly reduce their release to air, water and soil in order to minimize their adverse impacts on human health and the environment\n",
            "Target 29 has the following description: By 2030, ensure that people everywhere have the relevant information and awareness for sustainable development and lifestyles in harmony with nature\n",
            "Target 31 has the following description: Promote public procurement practices that are sustainable, in accordance with national policies and priorities\n",
            "Target 42 has the following description: Encourage companies, especially large and transnational companies, to adopt sustainable practices and to integrate sustainability information into their reporting cycle\n",
            "Target 48 has the following description: Develop and implement tools to monitor sustainable development impacts for sustainable tourism that creates jobs and promotes local culture and products\n",
            "Target 61 has the following description: By 2030, halve per capita global food waste at the retail and consumer levels and reduce food losses along production and supply chains, including post-harvest losses\n",
            "Target 72 has the following description: Rationalize inefficient fossil-fuel subsidies that encourage wasteful consumption by removing market distortions, in accordance with national circumstances, including by restructuring taxation and phasing out those harmful subsidies, where they exist, to reflect their environmental impacts, taking fully into account the specific needs and conditions of developing countries and minimizing the possible adverse impacts on their development in a manner that protects the poor and the affected communities\n",
            "Target 81 has the following description: Support developing countries to strengthen their scientific and technological capacity to move towards more sustainable patterns of consumption and production\n"
          ]
        }
      ],
      "source": [
        "# Find all target_ids that belong to goal 0\n",
        "target_ids = []\n",
        "for target_id, goal_id in target_idx_2_goal_idx.items():\n",
        "    if goal_id == goal_idx:\n",
        "        target_ids.append(target_id)\n",
        "\n",
        "# And now lets print all the descriptions of the targets that belong to goal 0\n",
        "for target_id in target_ids:\n",
        "    print(f'Target {target_id} has the following description: {target_idx_2_description[target_id]}')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<Axes: xlabel='goal'>"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjAAAAG0CAYAAAAsOB08AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAtr0lEQVR4nO3deXQUVd7/8U9nJSwJazaJEARZlM0IGFkGNUPEgDAy+qBIXABHnuCIcRBwGED0SMR9GzjOKKgDGHweVCCCBFDWCBINICKCgjBCB34gaUBJArm/Pzyph1bAIVTs3OT9OqfOoerequ+9aTr9SVV1t8cYYwQAAGCRoEAPAAAA4HwRYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArBMS6AFUlrKyMu3bt0/16tWTx+MJ9HAAAMB/wBijo0ePKj4+XkFBZz/PUm0DzL59+5SQkBDoYQAAgArYu3evmjZtetb2ahtg6tWrJ+mnH0BkZGSARwMAAP4TPp9PCQkJzuv42VTbAFN+2SgyMpIAAwCAZX7t9g9u4gUAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYJyTQAwiU5uNyKrzv7qw0F0cCAADOF2dgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrnFeAmTp1qrp06aJ69eopOjpaAwcO1Pbt2/369O7dWx6Px2+59957/frs2bNHaWlpql27tqKjozVmzBidPHnSr89HH32kK664QuHh4WrZsqVmzZpVsRkCAIBq57wCzMqVK5WRkaGPP/5Yubm5Ki0tVZ8+fXT8+HG/fiNGjND+/fudZdq0aU7bqVOnlJaWppKSEq1bt06vv/66Zs2apYkTJzp9du3apbS0NF1zzTUqKCjQ6NGjNXz4cH3wwQcXOF0AAFAdnNdXCSxZssRvfdasWYqOjlZ+fr569erlbK9du7ZiY2PPeIylS5fqiy++0LJlyxQTE6NOnTrp0Ucf1dixYzV58mSFhYVpxowZSkxM1NNPPy1Jatu2rdasWaNnn31Wqamp5ztHAABQzVzQPTBFRUWSpIYNG/ptnz17tho3bqzLL79c48eP1w8//OC05eXlqX379oqJiXG2paamyufzaevWrU6flJQUv2OmpqYqLy/vrGMpLi6Wz+fzWwAAQPVU4S9zLCsr0+jRo9W9e3ddfvnlzvbbbrtNzZo1U3x8vDZv3qyxY8dq+/btmj9/viTJ6/X6hRdJzrrX6z1nH5/Ppx9//FERERG/GM/UqVP1yCOPVHQ6AADAIhUOMBkZGfr888+1Zs0av+333HOP8+/27dsrLi5O1113nb7++mtdcsklFR/prxg/frwyMzOddZ/Pp4SEhEqrBwAAAqdCl5BGjRqlRYsW6cMPP1TTpk3P2bdbt26SpJ07d0qSYmNjVVhY6NenfL38vpmz9YmMjDzj2RdJCg8PV2RkpN8CAACqp/MKMMYYjRo1Su+8845WrFihxMTEX92noKBAkhQXFydJSk5O1pYtW3TgwAGnT25uriIjI9WuXTunz/Lly/2Ok5ubq+Tk5PMZLgAAqKbOK8BkZGToX//6l+bMmaN69erJ6/XK6/Xqxx9/lCR9/fXXevTRR5Wfn6/du3drwYIFSk9PV69evdShQwdJUp8+fdSuXTsNHTpUmzZt0gcffKAJEyYoIyND4eHhkqR7771X33zzjR566CF9+eWX+vvf/6558+bpgQcecHn6AADARucVYKZPn66ioiL17t1bcXFxzpKdnS1JCgsL07Jly9SnTx+1adNGDz74oAYNGqSFCxc6xwgODtaiRYsUHBys5ORk3X777UpPT9eUKVOcPomJicrJyVFubq46duyop59+Wv/85z95CzUAAJAkeYwxJtCDqAw+n09RUVEqKio64/0wzcflVPjYu7PSLmRoAADgLH7t9bsc34UEAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxzXgFm6tSp6tKli+rVq6fo6GgNHDhQ27dv9+tz4sQJZWRkqFGjRqpbt64GDRqkwsJCvz579uxRWlqaateurejoaI0ZM0YnT5706/PRRx/piiuuUHh4uFq2bKlZs2ZVbIYAAKDaOa8As3LlSmVkZOjjjz9Wbm6uSktL1adPHx0/ftzp88ADD2jhwoV6++23tXLlSu3bt0833XST037q1CmlpaWppKRE69at0+uvv65Zs2Zp4sSJTp9du3YpLS1N11xzjQoKCjR69GgNHz5cH3zwgQtTBgAAtvMYY0xFdz548KCio6O1cuVK9erVS0VFRWrSpInmzJmjP/7xj5KkL7/8Um3btlVeXp6uuuoqLV68WP369dO+ffsUExMjSZoxY4bGjh2rgwcPKiwsTGPHjlVOTo4+//xzp9bgwYN15MgRLVmy5D8am8/nU1RUlIqKihQZGfmL9ubjcio6be3OSqvwvgAA4Ox+7fW73AXdA1NUVCRJatiwoSQpPz9fpaWlSklJcfq0adNGF198sfLy8iRJeXl5at++vRNeJCk1NVU+n09bt251+px+jPI+5cc4k+LiYvl8Pr8FAABUTxUOMGVlZRo9erS6d++uyy+/XJLk9XoVFham+vXr+/WNiYmR1+t1+pweXsrby9vO1cfn8+nHH38843imTp2qqKgoZ0lISKjo1AAAQBVX4QCTkZGhzz//XG+99Zab46mw8ePHq6ioyFn27t0b6CEBAIBKElKRnUaNGqVFixZp1apVatq0qbM9NjZWJSUlOnLkiN9ZmMLCQsXGxjp9NmzY4He88ncpnd7n5+9cKiwsVGRkpCIiIs44pvDwcIWHh1dkOgAAwDLndQbGGKNRo0bpnXfe0YoVK5SYmOjXnpSUpNDQUC1fvtzZtn37du3Zs0fJycmSpOTkZG3ZskUHDhxw+uTm5ioyMlLt2rVz+px+jPI+5ccAAAA123mdgcnIyNCcOXP03nvvqV69es49K1FRUYqIiFBUVJSGDRumzMxMNWzYUJGRkbrvvvuUnJysq666SpLUp08ftWvXTkOHDtW0adPk9Xo1YcIEZWRkOGdQ7r33Xr300kt66KGHdPfdd2vFihWaN2+ecnIq/s4hAABQfZzXGZjp06erqKhIvXv3VlxcnLNkZ2c7fZ599ln169dPgwYNUq9evRQbG6v58+c77cHBwVq0aJGCg4OVnJys22+/Xenp6ZoyZYrTJzExUTk5OcrNzVXHjh319NNP65///KdSU1NdmDIAALDdBX0OTFXG58AAAGCf3+RzYAAAAAKBAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsE6FvswRFcOH5wEA4A7OwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1QgI9AFSu5uNyKrzv7qw0F0cCAIB7OAMDAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA65x1gVq1apf79+ys+Pl4ej0fvvvuuX/udd94pj8fjt1x//fV+fQ4fPqwhQ4YoMjJS9evX17Bhw3Ts2DG/Pps3b1bPnj1Vq1YtJSQkaNq0aec/OwAAUC2dd4A5fvy4OnbsqJdffvmsfa6//nrt37/fWebOnevXPmTIEG3dulW5ublatGiRVq1apXvuucdp9/l86tOnj5o1a6b8/Hw9+eSTmjx5sl555ZXzHS4AAKiGzvtzYPr27au+ffues094eLhiY2PP2LZt2zYtWbJEn3zyia688kpJ0osvvqgbbrhBTz31lOLj4zV79myVlJTotddeU1hYmC677DIVFBTomWee8Qs6AACgZqqUe2A++ugjRUdHq3Xr1ho5cqQOHTrktOXl5al+/fpOeJGklJQUBQUFaf369U6fXr16KSwszOmTmpqq7du36/vvvz9jzeLiYvl8Pr8FAABUT64HmOuvv15vvPGGli9frieeeEIrV65U3759derUKUmS1+tVdHS03z4hISFq2LChvF6v0ycmJsavT/l6eZ+fmzp1qqKiopwlISHB7akBAIAqwvWvEhg8eLDz7/bt26tDhw665JJL9NFHH+m6665zu5xj/PjxyszMdNZ9Ph8hBgCAaqrS30bdokULNW7cWDt37pQkxcbG6sCBA359Tp48qcOHDzv3zcTGxqqwsNCvT/n62e6tCQ8PV2RkpN8CAACqp0oPMP/+97916NAhxcXFSZKSk5N15MgR5efnO31WrFihsrIydevWzemzatUqlZaWOn1yc3PVunVrNWjQoLKHDAAAqrjzDjDHjh1TQUGBCgoKJEm7du1SQUGB9uzZo2PHjmnMmDH6+OOPtXv3bi1fvlwDBgxQy5YtlZqaKklq27atrr/+eo0YMUIbNmzQ2rVrNWrUKA0ePFjx8fGSpNtuu01hYWEaNmyYtm7dquzsbD3//PN+l4gAAEDNdd4BZuPGjercubM6d+4sScrMzFTnzp01ceJEBQcHa/Pmzbrxxht16aWXatiwYUpKStLq1asVHh7uHGP27Nlq06aNrrvuOt1www3q0aOH32e8REVFaenSpdq1a5eSkpL04IMPauLEibyFGgAASKrATby9e/eWMeas7R988MGvHqNhw4aaM2fOOft06NBBq1evPt/hAQCAGoDvQgIAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALBOSKAHgOqp+bicCu23OyvN5ZEAAKojzsAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOvwSbyoNvj0XwCoOTgDAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOrwLCbgAFX3nk8S7nwDgQnAGBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxz3gFm1apV6t+/v+Lj4+XxePTuu+/6tRtjNHHiRMXFxSkiIkIpKSnasWOHX5/Dhw9ryJAhioyMVP369TVs2DAdO3bMr8/mzZvVs2dP1apVSwkJCZo2bdr5zw4AAFRL5x1gjh8/ro4dO+rll18+Y/u0adP0wgsvaMaMGVq/fr3q1Kmj1NRUnThxwukzZMgQbd26Vbm5uVq0aJFWrVqle+65x2n3+Xzq06ePmjVrpvz8fD355JOaPHmyXnnllQpMEQAAVDch57tD37591bdv3zO2GWP03HPPacKECRowYIAk6Y033lBMTIzeffddDR48WNu2bdOSJUv0ySef6Morr5Qkvfjii7rhhhv01FNPKT4+XrNnz1ZJSYlee+01hYWF6bLLLlNBQYGeeeYZv6BzuuLiYhUXFzvrPp/vfKcGAAAs4eo9MLt27ZLX61VKSoqzLSoqSt26dVNeXp4kKS8vT/Xr13fCiySlpKQoKChI69evd/r06tVLYWFhTp/U1FRt375d33///RlrT506VVFRUc6SkJDg5tQAAEAV4mqA8Xq9kqSYmBi/7TExMU6b1+tVdHS0X3tISIgaNmzo1+dMxzi9xs+NHz9eRUVFzrJ3794LnxAAAKiSzvsSUlUVHh6u8PDwQA8DAAD8Blw9AxMbGytJKiws9NteWFjotMXGxurAgQN+7SdPntThw4f9+pzpGKfXAAAANZerASYxMVGxsbFavny5s83n82n9+vVKTk6WJCUnJ+vIkSPKz893+qxYsUJlZWXq1q2b02fVqlUqLS11+uTm5qp169Zq0KCBm0MGAAAWOu8Ac+zYMRUUFKigoEDSTzfuFhQUaM+ePfJ4PBo9erQee+wxLViwQFu2bFF6erri4+M1cOBASVLbtm11/fXXa8SIEdqwYYPWrl2rUaNGafDgwYqPj5ck3XbbbQoLC9OwYcO0detWZWdn6/nnn1dmZqZrEwcAAPY673tgNm7cqGuuucZZLw8Vd9xxh2bNmqWHHnpIx48f1z333KMjR46oR48eWrJkiWrVquXsM3v2bI0aNUrXXXedgoKCNGjQIL3wwgtOe1RUlJYuXaqMjAwlJSWpcePGmjhx4lnfQg0AAGqW8w4wvXv3ljHmrO0ej0dTpkzRlClTztqnYcOGmjNnzjnrdOjQQatXrz7f4QEAgBqA70ICAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALBOtfkuJKCmaD4up8L77s5Kc3EkABA4nIEBAADWIcAAAADrEGAAAIB1uAcGwK/ivhsAVQ1nYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKwTEugBAMDZNB+XU6H9dmeluTwSAFUNZ2AAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYJyTQAwCAqqT5uJwK7bc7K83lkQA4F9fPwEyePFkej8dvadOmjdN+4sQJZWRkqFGjRqpbt64GDRqkwsJCv2Ps2bNHaWlpql27tqKjozVmzBidPHnS7aECAABLVcoZmMsuu0zLli37vyIh/1fmgQceUE5Ojt5++21FRUVp1KhRuummm7R27VpJ0qlTp5SWlqbY2FitW7dO+/fvV3p6ukJDQ/X4449XxnABAIBlKiXAhISEKDY29hfbi4qK9Oqrr2rOnDm69tprJUkzZ85U27Zt9fHHH+uqq67S0qVL9cUXX2jZsmWKiYlRp06d9Oijj2rs2LGaPHmywsLCKmPIAADAIpVyE++OHTsUHx+vFi1aaMiQIdqzZ48kKT8/X6WlpUpJSXH6tmnTRhdffLHy8vIkSXl5eWrfvr1iYmKcPqmpqfL5fNq6detZaxYXF8vn8/ktAACgenI9wHTr1k2zZs3SkiVLNH36dO3atUs9e/bU0aNH5fV6FRYWpvr16/vtExMTI6/XK0nyer1+4aW8vbztbKZOnaqoqChnSUhIcHdiAACgynD9ElLfvn2df3fo0EHdunVTs2bNNG/ePEVERLhdzjF+/HhlZmY66z6fjxADAEA1VemfA1O/fn1deuml2rlzp2JjY1VSUqIjR4749SksLHTumYmNjf3Fu5LK1890X0258PBwRUZG+i0AAKB6qvQAc+zYMX399deKi4tTUlKSQkNDtXz5cqd9+/bt2rNnj5KTkyVJycnJ2rJliw4cOOD0yc3NVWRkpNq1a1fZwwUAABZw/RLSX/7yF/Xv31/NmjXTvn37NGnSJAUHB+vWW29VVFSUhg0bpszMTDVs2FCRkZG67777lJycrKuuukqS1KdPH7Vr105Dhw7VtGnT5PV6NWHCBGVkZCg8PNzt4QIAAAu5HmD+/e9/69Zbb9WhQ4fUpEkT9ejRQx9//LGaNGkiSXr22WcVFBSkQYMGqbi4WKmpqfr73//u7B8cHKxFixZp5MiRSk5OVp06dXTHHXdoypQpbg8VAABYyvUA89Zbb52zvVatWnr55Zf18ssvn7VPs2bN9P7777s9NAAAUE3wZY4AAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOiGBHgAA1HTNx+VUaL/dWWkujwSwBwEGAGqgioYmieCEqoFLSAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArMO7kAAAvwne+QQ3cQYGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKzDVwkAAKotvr6g+iLAAADgsooGJ0LTf45LSAAAwDqcgQEAoBqoaWd9OAMDAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1gkJ9ADO5eWXX9aTTz4pr9erjh076sUXX1TXrl0DPSwAACCp+bicCu+7OyvtgmpX2TMw2dnZyszM1KRJk/Tpp5+qY8eOSk1N1YEDBwI9NAAAEGBVNsA888wzGjFihO666y61a9dOM2bMUO3atfXaa68FemgAACDAquQlpJKSEuXn52v8+PHOtqCgIKWkpCgvL++M+xQXF6u4uNhZLyoqkiT5fL4z9i8r/qHC4zvbMX9NTal5IXVrSs0LqVtTal5IXWpWXs0LqVtTal5I3ZpS81x1y7cbY859AFMFfffdd0aSWbdund/2MWPGmK5du55xn0mTJhlJLCwsLCwsLNVg2bt37zmzQpU8A1MR48ePV2ZmprNeVlamw4cPq1GjRvJ4PP/xcXw+nxISErR3715FRkZWxlCrTF1qUtPWutSsXjUDVZeaVbOmMUZHjx5VfHz8OftVyQDTuHFjBQcHq7Cw0G97YWGhYmNjz7hPeHi4wsPD/bbVr1+/wmOIjIz8TZ+8gaxLTWraWpea1atmoOpSs+rVjIqK+tU+VfIm3rCwMCUlJWn58uXOtrKyMi1fvlzJyckBHBkAAKgKquQZGEnKzMzUHXfcoSuvvFJdu3bVc889p+PHj+uuu+4K9NAAAECAVdkA81//9V86ePCgJk6cKK/Xq06dOmnJkiWKiYmp1Lrh4eGaNGnSLy5HVbZA1KUmNW2tS83qVTNQdalpd02PMb/2PiUAAICqpUreAwMAAHAuBBgAAGAdAgwAALAOAQYAAFiHAAMAqJF4D4vdquzbqAFUTfv379f06dO1Zs0a7d+/X0FBQWrRooUGDhyoO++8U8HBwYEeIvAfCQ8P16ZNm9S2bdtADwUVUOPPwNx3331avXp1oIfxm/jxxx+1Zs0affHFF79oO3HihN54440AjMpdn376qXbt2uWsv/nmm+revbsSEhLUo0cPvfXWWwEcnf02btyotm3b6v3331dpaal27NihpKQk1alTR3/5y1/Uq1cvHT161PW627Zt08yZM/Xll19Kkr788kuNHDlSd999t1asWOF6PfyfvXv36u6773b9uC+99JLS09Od5+Sbb76pdu3aqU2bNnr44Yd18uRJ12plZmaecTl16pSysrKc9ero+PHjmjlzpv7617/qpZde0qFDhwI9JPe48vXRFvN4PCYoKMi0atXKZGVlmf379//mYygrKzMrVqwwr7zyilm4cKEpKSlxvcb27dtNs2bNnPn26tXL7Nu3z2n3er0mKCjI9bq/xuv1mkceecS143Xo0MHk5uYaY4z5xz/+YSIiIsyf//xnM336dDN69GhTt25d8+qrr7pW73T/7//9P7NixQpz6NAhY4wxBw8eNFlZWeaRRx4xX3zxRaXUNMaYvXv3mqNHj/5ie0lJiVm5cqWrtbp3724mT57srL/55pumW7duxhhjDh8+bDp16mT+/Oc/u1pz8eLFJiwszDRs2NDUqlXLLF682DRp0sSkpKSYa6+91gQHB5vly5e7WvNcEhMTzVdffVUpx37qqafM7t27K+XYFVVQUOD674ZHH33U1KtXzwwaNMjExsaarKws06hRI/PYY4+Zxx9/3DRp0sRMnDjRtXoej8d06tTJ9O7d22/xeDymS5cupnfv3uaaa65xrd7piouLTXZ2thk9erQZPHiwGTx4sBk9erSZN2+eKS4udr1e27Ztnd9Be/bsMc2bNzdRUVGmS5cupmHDhiY6Otp88803rtfdu3evOXjwoLO+atUqc9ttt5kePXqYIUOGmHXr1rlekwDj8Zhly5aZ+++/3zRu3NiEhoaaG2+80SxcuNCcOnWqUmr27dvXHDlyxBhjzKFDh0y3bt2Mx+MxTZo0MUFBQaZNmzbmwIEDrtYcOHCgSUtLMwcPHjQ7duwwaWlpJjEx0Xz77bfGmMAFGLd/OUZERDgvAJ07dzavvPKKX/vs2bNNu3btXKtXbv369SYqKsp4PB7ToEEDs3HjRpOYmGhatWplLrnkEhMREWHy8/Ndrblv3z7TpUsXExQUZIKDg83QoUP9gkxlPKYRERHm66+/dtZPnTplQkNDjdfrNcYYs3TpUhMfH+9qzeTkZPPXv/7VGGPM3LlzTYMGDczDDz/stI8bN878/ve/d7WmMcY8//zzZ1yCg4PN+PHjnXU3eTweExwcbFJSUsxbb71VKS9wP/fee++dc3n22Wdd/390ySWXmP/93/81xvz0OyA4ONj861//ctrnz59vWrZs6Vq9qVOnmsTExF8E3ZCQELN161bX6vzcjh07TIsWLUytWrXM7373O3PLLbeYW265xfzud78ztWrVMi1btjQ7duxwtabH4zGFhYXGGGOGDBlirr76auf15ujRoyYlJcXceuutrtY0xpiuXbuahQsXGmOMeffdd01QUJC58cYbzdixY80f/vAHExoa6rS7hQBz2oNdUlJisrOzTWpqqgkODjbx8fHm4YcfrtT/YCNHjjTt2rVzEvHevXtNUlKSuffee12tGR0dbTZv3uysl5WVmXvvvddcfPHF5uuvv660ALNp06ZzLtnZ2a7WbdSokdm4caMx5qc5FxQU+LXv3LnTREREuFavXEpKihk+fLjx+XzmySefNE2bNjXDhw932u+66y4zcOBAV2ump6ebbt26mU8++cTk5uaapKQkc+WVV5rDhw8bY34KMB6Px9WazZo1M2vWrHHW9+3bZzwej/nhhx+MMcbs2rXL1KpVy9WakZGRznPw1KlTJiQkxHz66adO+5YtW0xMTIyrNY356XnatGlT07x5c7/F4/GYiy66yDRv3twkJia6XnPmzJlmwIABJjQ01DRq1Mjcf//9ZsuWLa7W+XnNoKAg4/F4zrpURhAu/+PJGGNCQ0PN559/7qzv3r3b1K5d29WaGzZsMJdeeql58MEHnbPclR1gUlJSzIABA0xRUdEv2oqKisyAAQNMnz59XK15+utLixYtzNKlS/3a165daxISElytaYwxderUcV7HunXrZrKysvzaX3zxRdO5c2dXaxJgTnuwT/ftt9+aSZMmmWbNmrn+5D29ZuvWrc17773n175s2TLXfzHWq1fvjJcxMjIyTNOmTc2qVasqJcCc65dj+XY3695+++1m2LBhxhhjbr75ZjNhwgS/9scff9y0b9/etXrlGjRo4Px8S0pKTFBQkFm/fr3Tnp+fby666CJXa8bHx/vVOHHihOnfv7/p1KmTOXToUKWE0vvvv99cfvnlZvHixWbFihXmmmuuMb1793balyxZYi655BJXa0ZGRpqdO3c663Xr1vU7C7R7927XQ5MxxvzpT38ynTp1+sXzpjJf9E7/3VBYWGieeOIJ06ZNGxMUFGS6dOliXnnlFePz+VytGR8fb959992ztn/22Weu/z9KTEw0ixcvNsYY89VXX5mgoCAzb948pz0nJ8c0b97c1ZrG/HQGIj093XTo0MFs2bLFhIaGVmqAiYiIOGf43Lx5s+t/UHk8HucMfnx8/C/qV9bzJSoqymzatMkY89Mfj+X/Lrdz507XQykB5iwBplxZWdkvEqwbNcv/g0VHR/v95WHMT//BwsPDXa3ZpUsX88Ybb5yxLSMjw9SvX79SAkyjRo3Mq6++anbv3n3GJScnx9W63333nWnevLnp1auXyczMNBEREaZHjx5mxIgRplevXiYsLMzk5OS4Vq9cnTp1zK5du5z1n7/Ifvvtt67/0qhTp84v7sUoLS01AwcONB06dDCbN292/TE9evSoueWWW0xISIjxeDzm6quv9rue/sEHH/i9ELmhQ4cOzoudMT+dcSktLXXWV61a5XrgLzd//nyTkJBgXnzxRWfbbxVgTrdq1Spzxx13mDp16pg6deq4WrN///7mb3/721nbCwoKXD+TN2HCBNOkSRMzfPhwk5iYaMaNG2cuvvhiM336dDNjxgyTkJBgHnjgAVdrnm7u3LkmJibGBAUFVWqAiYuLO+dlkwULFpi4uDhXa3o8HtO+fXvTuXNnU7duXfM///M/fu0rV650/Y8pY4y58cYbzbhx44wxxqSmpv7i8uo//vEP06pVK1dr1vi3UTdr1uycb/v0eDz6/e9/73rdO++8U+Hh4SotLdWuXbt02WWXOW1er1f169d3td4f/vAHzZ07V0OHDv1F20svvaSysjLNmDHD1ZqSlJSUpH379qlZs2ZnbD9y5Iirn8UQHx+vzz77TFlZWVq4cKGMMdqwYYP27t2r7t27a+3atbryyitdq1cuISFB33zzjZo3by5JeuuttxQXF+e079+/X40bN3a1ZosWLbR582a1atXK2RYSEqK3335bN998s/r16+dqPUmqW7eusrOzdeLECZ08eVJ169b1a+/Tp4/rNUeOHKlTp04565dffrlf++LFi3Xttde6Xlf66XnTtWtXpaenKycnRzNnzqyUOuU8Hs8Zt/fs2VM9e/bUCy+8oOzsbFdrjhkzRsePHz9re8uWLfXhhx+6WvORRx5RRESE8vLyNGLECI0bN04dO3bUQw89pB9++EH9+/fXo48+6mrN0w0ePFg9evRQfn7+WX83uWH48OFKT0/X3/72N1133XWKiYmRJBUWFmr58uV67LHHdN9997lac9KkSX7rP3+OLly4UD179nS1piRlZWWpZ8+e2rdvn3r06KG//vWv+uSTT9S2bVtt375d2dnZ7r/GuBqH8B+58847/Zbs7Gy/9jFjxpjU1NQAjc5d8+fPN2+++eZZ2w8fPmxmzZr1G46ockyePNnMnTv3rO0PP/ywuemmm1yt+dBDD531+nlpaam58cYbXf/LuaYqKyszjz/+uImNjTXBwcG/+RkY2CsrK8vExcU5l8vLL53HxcWZJ554ItDDc9XOnTvN4MGDTb169ZxbBUJDQ83VV19t3nnnHdfreYzhowirmuPHjys4OFi1atUK9FDgkh9++EHBwcEKDw937ZgnT57UDz/8oMjIyLO2f/fdd5X6F2ZNk5+frzVr1ig9PV0NGjQI9HBgkV27dsnr9UqSYmNjlZiYGOARVR5jjA4cOKCysjI1btxYoaGhlVKnxn+QXVV0+PBh/fd//3egh/GbqKwPyapqDh06pJEjR7p6zJCQkLOGF+mny1aPPPKIqzVruqSkJN1///1q0KBBQP7v1pTnS3WUmJio5ORkJScnO+Gluj6eHo9HMTExiouLc8JLZcyVMzBV0KZNm3TFFVf4XfevrmrKXAMxz5rysw0UHlNcqJr0eFbGXGv8TbyBsGDBgnO2f/PNN7/RSCpfTZlrIOZZU362gcJjigtVkx7PQMyVMzABEBQUJI/Hc85333g8nmqRymvKXAMxz5rysw0UHlNcqJr0eAbk+eLakfAfi4uL0/z581VWVnbG5dNPPw30EF1TU+YaiHnWlJ9toPCY4kLVpMczEHMlwARAUlKS8vPzz9r+aynWJjVlroGYZ0352QYKjykuVE16PAMxV+6BCYBAfHBUoNSUuQZinjXlZxsoPKa4UDXp8QzEXLkHBgAAWIdLSAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAA6BG6N27t0aPHh3oYQBwCQEGAABYhwADAACsQ4ABEHBHjx7VkCFDVKdOHcXFxenZZ5/1u+Tz/fffKz09XQ0aNFDt2rXVt29f7dixw9n/0KFDuvXWW3XRRRepdu3aat++vebOnRug2QD4LRBgAARcZmam1q5dqwULFig3N1erV6/2++6UO++8Uxs3btSCBQuUl5cnY4xuuOEGlZaWSpJOnDihpKQk5eTk6PPPP9c999yjoUOHasOGDYGaEoBKxifxAgioo0ePqlGjRpozZ47++Mc/SpKKiooUHx+vESNGKCMjQ5deeqnWrl2rq6++WtJPZ1wSEhL0+uuv6+abbz7jcfv166c2bdroqaeekvTTTbydOnXSc88995vMC0Dl4ruQAATUN998o9LSUnXt2tXZFhUVpdatW0uStm3bppCQEHXr1s1pb9SokVq3bq1t27ZJkk6dOqXHH39c8+bN03fffaeSkhIVFxerdu3av+1kAPxmCDAArPfkk0/q+eef13PPPaf27durTp06Gj16tEpKSgI9NACVhHtgAARUixYtFBoaqk8++cTZVlRUpK+++kqS1LZtW508eVLr16932g8dOqTt27erXbt2kqS1a9dqwIABuv3229WxY0e1aNHC2R9A9USAARBQ9erV0x133KExY8boww8/1NatWzVs2DAFBQXJ4/GoVatWGjBggEaMGKE1a9Zo06ZNuv3223XRRRdpwIABkqRWrVopNzdX69at07Zt2/SnP/1JhYWFAZ4ZgMpEgAEQcM8884ySk5PVr18/paSkqHv37mrbtq1q1aolSZo5c6aSkpLUr18/JScnyxij999/X6GhoZKkCRMm6IorrlBqaqp69+6t2NhYDRw4MIAzAlDZeBcSgCrn+PHjuuiii/T0009r2LBhgR4OgCqIm3gBBNxnn32mL7/8Ul27dlVRUZGmTJkiSc4lIgD4OQIMgCrhqaee0vbt2xUWFqakpCStXr1ajRs3DvSwAFRRXEICAADW4SZeAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6/x+cv8HHCueJ/gAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Lets also plot the histogram of goals in the dataset\n",
        "df_full['goal'].value_counts().plot(kind='bar')\n",
        "# ! Todo: also maybe add the contigency here to show that many possible predictions are possible?"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a name=\"simple-prompt\"></a>\n",
        "## Using prompting to classify the text into the correct SDG goal.\n",
        "- In this section we will set up the classification task that we wish to perform.\n",
        "- We will feed in the text from the NDC and have the model predict the corresponding SDG goal.\n",
        "- Unlike in the previous tutorial where we fine-tuned a classifier on supervised data, here we will adopt the strategy of prompting a Large Language Model (LLM) to perform this in a zero-shot and few-shot manner.\n",
        "- First let's do a quick review so that we can understand what prompting is and how it works.\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Large Language Models (LLMs) and Prompting\n",
        "- In the previous tutorial, we fine-tuned a classification head on top of a pretrained transformer model.\n",
        "- Given a sentence the model outputted a vector of dimension 'd' and we added new parameter weights (classification head) to map this vector to a vector of dimension 'k' where 'k' is the number of classes.\n",
        "- Since these classifier's parameters are randomly initialized, we needed to train them on supervised data in order to learn the correct mapping.\n",
        "- This may be limiting if we have too little labeled data or its too expensive to label data.\n",
        "- What if instead of introducing new parameters we could use the same underlying transformer model to also perform the classification task?\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Enter Prompting\n",
        "- LLMs are typically pretrained on large amounts of unlabeled text with a self-supervised objective: predicting the next word.\n",
        "- The initial idea of prompting can then be thought as: \"How can we change this classification task such that it falls naturally to next word prediction?\"\n",
        "- Suppose we start with a sentiment classification sentence like \"I loved this movie, it was a blast!\" where the true label is \"positive\".\n",
        "- We can then add a prompt or a template of the form \"{Movie Review}. This movie is {sentiment}\" \n",
        "    - input to model: \"I loved this movie, it was a blast! This movie is\"\n",
        "    - and we can compare the next word prediction probability of the label \"positive\" versus \"negative\".\n",
        "- Whichever is higher is the label we choose.\n",
        "- This is a simple example of prompting, where we reframed sentiment classification to fit with the next word prediction which allows us to use the same pretrained model without any additional parameters.\n",
        "\n",
        "### Current LLMs and Prompting\n",
        "- The simple example above gives intuition for simple prompting but in practice current LLMs are much more flexible with the types of prompts that can be used.\n",
        "- They have been trained on extra datasets for following instructions so that they can perform tasks where the answers don't have to be mapped directly to the next word prediction.\n",
        "- In this first section, we are going to build a simple instruction prompt which will describe our SDG Goal classification task to the model.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "9jVjv8etAIiQ"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import random\n",
        "\n",
        "# set seeds for reproducibility\n",
        "np.random.seed(42)\n",
        "random.seed(42)\n",
        "\n",
        "\n",
        "# lets setup some global variables\n",
        "\n",
        "# for goal prediction this will be the number of classes \n",
        "N_PRED_CLASSES = len(goal_idx_2_goal_number)\n",
        "# this will signify to the model the start of the input text\n",
        "INPUT_MARKER = 'Input Text'\n",
        "\n",
        "# This dictionary just contains the names of the columns to slightly longer names that are easier to understand.\n",
        "column_names = {'goal': 'Sustainable Development Goal', 'target': 'Subgoal', 'status': 'Status', 'sector': 'Sector', 'response': 'Climate Response', 'infotype': 'Information Type'}\n",
        "#"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Constructing the Prompt\n",
        "- We are going to construct the prompt in two parts: instructions at the top and the input text to classify at the bottom."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Given the following Input Text predict the Sustainable Development Goal (goal) out of the following 17 options:\n",
            "Sustainable Development Goal\n",
            "1: End poverty in all its forms everywhere\n",
            "2: End hunger, achieve food security and improved nutrition and promote sustainable agriculture\n",
            "3: Ensure healthy lives and promote well-being for all at all ages\n",
            "4: Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all\n",
            "5: Achieve gender equality and empower all women and girls\n",
            "6: Ensure availability and sustainable management of water and sanitation for all\n",
            "7: Ensure access to affordable, reliable, sustainable and modern energy for all\n",
            "8: Promote sustained, inclusive and sustainable economic growth, full and productive employment and decent work for all\n",
            "9: Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation\n",
            "10: Reduce inequality within and among countries\n",
            "11: Make cities and human settlements inclusive, safe, resilient and sustainable\n",
            "12: Ensure sustainable consumption and production patterns\n",
            "13: Take urgent action to combat climate change and its impacts\n",
            "14: Conserve and sustainably use the oceans, seas and marine resources for sustainable development\n",
            "15: Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss\n",
            "16: Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels\n",
            "17: Strengthen the means of implementation and revitalize the global partnership for sustainable development\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# for now we will skip the special_token as well as the what the \"context\" column is.\n",
        "SPECIAL_TOKEN = \"**\"\n",
        "def build_instruction(pred_column, obs_column):\n",
        "    ''' Args:\n",
        "            pred_column (str): the column that we are predicting\n",
        "            obs_column (str): the column that we are observing (e.g. text, context) \n",
        "        Returns:\n",
        "            prompt (str): the prompt that will be used to query the model\n",
        "    \n",
        "    '''\n",
        "    if obs_column == 'text':\n",
        "        # describe what the model's input, task and class options are.\n",
        "        prompt = f'Given the following {INPUT_MARKER} predict the {column_names[pred_column]} ({pred_column}) out of the following {N_PRED_CLASSES} options:\\n'\n",
        "    elif obs_column == 'context':\n",
        "        prompt = f'Given the following {INPUT_MARKER}, marked with {SPECIAL_TOKEN} at the start and end. Predict the {column_names[pred_column]} ({pred_column}) out of the following {N_PRED_CLASSES} options:\\n'\n",
        "        \n",
        "    # for now we will assume that we are predicting the goal column\n",
        "    assert pred_column == 'goal'\n",
        "    goal_number_2_goal_idx = {v: k for k, v in goal_idx_2_goal_number.items()}\n",
        "    \n",
        "    # Here we are building a dictionary that maps the goal number to the goal description\n",
        "    goal_number_2_goal_description = {k:goal_idx_2_description[v] for k, v in goal_number_2_goal_idx.items()}\n",
        "    \n",
        "    # we sort this according to the goal number\n",
        "    sorted_keys = sorted(int(k) for k in list(goal_number_2_goal_description.keys()))\n",
        "    sorted_keys = [str(k) for k in sorted_keys]\n",
        "    \n",
        "    # build a list of strings where each string is the goal number and the goal description\n",
        "    values = [f'{key}:{goal_number_2_goal_description[key]}' for key in sorted_keys]\n",
        "    name = column_names[pred_column]\n",
        "    # and we join each goal/description with a new line to the intruction\n",
        "    prompt += name + '\\n' +\"\\n\".join(values) + \"\\n\\n\"\n",
        "    return prompt\n",
        "prompt_example = build_instruction(pred_column='goal', obs_column='text')\n",
        "print(prompt_example)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Given the following Input Text predict the Sustainable Development Goal (goal) out of the following 17 options:\n",
            "Sustainable Development Goal\n",
            "1: End poverty in all its forms everywhere\n",
            "2: End hunger, achieve food security and improved nutrition and promote sustainable agriculture\n",
            "3: Ensure healthy lives and promote well-being for all at all ages\n",
            "4: Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all\n",
            "5: Achieve gender equality and empower all women and girls\n",
            "6: Ensure availability and sustainable management of water and sanitation for all\n",
            "7: Ensure access to affordable, reliable, sustainable and modern energy for all\n",
            "8: Promote sustained, inclusive and sustainable economic growth, full and productive employment and decent work for all\n",
            "9: Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation\n",
            "10: Reduce inequality within and among countries\n",
            "11: Make cities and human settlements inclusive, safe, resilient and sustainable\n",
            "12: Ensure sustainable consumption and production patterns\n",
            "13: Take urgent action to combat climate change and its impacts\n",
            "14: Conserve and sustainably use the oceans, seas and marine resources for sustainable development\n",
            "15: Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss\n",
            "16: Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels\n",
            "17: Strengthen the means of implementation and revitalize the global partnership for sustainable development\n",
            "\n",
            "\n",
            "Input Text: Encourage and support the use of renewable energy in agricultural irrigation and in drinking water supply\n",
            "goal:\n"
          ]
        }
      ],
      "source": [
        "def build_single_prompt(row, pred_column, obs_column, test_input_only=False):\n",
        "    ''' Args:\n",
        "        row (pd.Series): a row from the dataframe\n",
        "        obs_column (str): the column that we are observing (e.g. text, context)\n",
        "        test_input_only (bool):\n",
        "            - if True, only return the input text\n",
        "                - This is used for the test example.\n",
        "            - if False, return the input text and the correct answer\n",
        "                - This will be for in-context learning\n",
        "        Returns:\n",
        "            prompt (str): the second portion of the prompt\n",
        "    '''\n",
        "    prompt = f'\\n{INPUT_MARKER}: {row[obs_column]}\\n'\n",
        "    if test_input_only:\n",
        "        return prompt\n",
        "    \n",
        "    value = row[pred_column]\n",
        "    prompt += f'{pred_column}:{value}\\n'\n",
        "    prompt += '\\n'\n",
        "    return prompt\n",
        "\n",
        "\n",
        "# sample a random row from the dataframe to use as a test example\n",
        "random_test_row = df_full.sample(1).iloc[0]\n",
        "test_input = build_single_prompt(random_test_row, pred_column='goal', obs_column='text', test_input_only=True)\n",
        "\n",
        "prompt_example += test_input\n",
        "prompt_example += f'{\"goal\"}:'\n",
        "# here we have the full instruction and prompt for the model\n",
        "print(prompt_example)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Let's get model predictions for this test example now"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "1\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "\n",
        "# To use the model, you will need to sign up for an OpenAI Account (http://platform.openai.com/signup) and get an API key pasted below.\n",
        "os.environ[\"OPENAI_API_KEY\"] = \"INSERT_YOUR_KEY_HERE\"\n",
        "openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n",
        "\n",
        "# so we will use this particular model\n",
        "MODEL_NAME = \"text-davinci-003\"\n",
        "\n",
        "def get_model_response(test_prompt, pay=False):\n",
        "  ''' test_prompt (str)\n",
        "        - The full text prompt to send to the API for completion.\n",
        "      pay (bool): \n",
        "        - If False, return a dummy response. If True, send the prompt to the model API \n",
        "        - This simple flag to let us test the function without calling the API which costs money.\n",
        "      Returns:\n",
        "        response (dict): the response from the API or a dummy response if Pay is False\n",
        "  '''\n",
        "  if not pay:\n",
        "    # this is a dummy response that always returns the class 3\n",
        "    response = {'choices':[{'text':'3'}]}\n",
        "  else:\n",
        "    response = openai.Completion.create(\n",
        "      model=MODEL_NAME,\n",
        "      prompt=test_prompt,\n",
        "      temperature=0,\n",
        "      max_tokens=4,\n",
        "      top_p=1.0,\n",
        "      frequency_penalty=0.0,\n",
        "      presence_penalty=0.0\n",
        "    )\n",
        "  return response\n",
        "\n",
        "\n",
        "# For now we can just test the function, but you can see what happens by setting pay=True\n",
        "response = get_model_response(prompt_example, pay=False)\n",
        "example_pred = response['choices'][0]['text']\n",
        "print(example_pred)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "128 43 43\n",
            "train: (9611, 13)\tval: (2507, 13)\ttest: (2101, 13)\n"
          ]
        }
      ],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "def make_new_splits(df):\n",
        "    '''We want to make sure that we split into train/val/test by documents and not by sentences. This helps us ensure that we aren't overfitting to a particular document/country.\n",
        "    Suppose (hyperbolically) that each document only had one SDG class. Then if we split by sentence, the model may overfit by first predicting which country the sentence is from, and then predicting the SDG goal based on that country. This is undesirable and so it is better to split by documents.\n",
        "    Args:\n",
        "        df (pd.DataFrame): the full dataframe\n",
        "    Returns:\n",
        "        df_train (pd.DataFrame): the training dataframe\n",
        "        df_val (pd.DataFrame): the validation dataframe\n",
        "        df_test (pd.DataFrame): the test dataframe\n",
        "    '''    \n",
        "    seed = 42\n",
        "    filenames = df['filename'].unique()\n",
        "    train_filenames, test_filenames = train_test_split(filenames, test_size=0.4, random_state=seed)\n",
        "    val_filenames, test_filenames = train_test_split(test_filenames, test_size=0.5, random_state=seed)\n",
        "    print(len(train_filenames), len(val_filenames), len(test_filenames))\n",
        "    df_train = df[df['filename'].isin(train_filenames)]\n",
        "    df_val = df[df['filename'].isin(val_filenames)]\n",
        "    df_test = df[df['filename'].isin(test_filenames)]\n",
        "    print(f'train: {df_train.shape}\\tval: {df_val.shape}\\ttest: {df_test.shape}')\n",
        "    return df_train, df_val, df_test\n",
        "\n",
        "df_train, df_val, df_test = make_new_splits(df_full)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a name=\"icl-prompt\"></a>\n",
        "## In-Context Learning Prompts\n",
        "- Before running and evaluating the model's performance, let's improve our prompt by adding in-context learning (ICL) examples.\n",
        "- So after we describe the instructions for the task, we will now add a few input/output examples that we want the model to match.\n",
        "- We will use a new function \"in_context_prompt\" which is simply a wrapper around the previous functions we used."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wJn5nUJIAIiQ",
        "outputId": "9e67c5bd-cc05-4644-c255-81efca59bf91"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Given the following Input Text predict the Sustainable Development Goal (goal) out of the following 17 options:\n",
            "Sustainable Development Goal\n",
            "1: End poverty in all its forms everywhere\n",
            "2: End hunger, achieve food security and improved nutrition and promote sustainable agriculture\n",
            "3: Ensure healthy lives and promote well-being for all at all ages\n",
            "4: Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all\n",
            "5: Achieve gender equality and empower all women and girls\n",
            "6: Ensure availability and sustainable management of water and sanitation for all\n",
            "7: Ensure access to affordable, reliable, sustainable and modern energy for all\n",
            "8: Promote sustained, inclusive and sustainable economic growth, full and productive employment and decent work for all\n",
            "9: Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation\n",
            "10: Reduce inequality within and among countries\n",
            "11: Make cities and human settlements inclusive, safe, resilient and sustainable\n",
            "12: Ensure sustainable consumption and production patterns\n",
            "13: Take urgent action to combat climate change and its impacts\n",
            "14: Conserve and sustainably use the oceans, seas and marine resources for sustainable development\n",
            "15: Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss\n",
            "16: Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels\n",
            "17: Strengthen the means of implementation and revitalize the global partnership for sustainable development\n",
            "\n",
            "\n",
            "Input Text: C4.</strong> Promote actions to prevent the establishment, control and eradication of invasive species, diseases and pests, whose impacts are exacerbated by the effects of climate change\n",
            "goal:13\n",
            "\n",
            "\n",
            "Input Text: Power generation from biomass energy: 160 MW of installed power to deliver to the power grid by 2025 (4% of the installed power from the SIN)(*).\n",
            "goal:7\n",
            "\n",
            "\n",
            "Input Text: Enhancing the adaptive capacity of rural economies and natural resources to climate change through the management and protection of land based natural resources and agricultural production systems.\n",
            "goal:12\n",
            "\n",
            "\n",
            "Input Text: Drafting of a regional spatial planning scheme for the different regions with the involvement of local communities\n",
            "goal:16\n",
            "\n",
            "\n",
            "Input Text: nature-based solutions play an important role to solve global challenges such as biodiversity loss and ecosystems degradation, poverty, hunger, health, water scarcity and drought, gender inequality, disaster risk reduction and climate change\n",
            "goal:2\n",
            "\n",
            "\n",
            "Input Text: Cambodia’s main national development priority, enshrined in the National Strategic Development Plan (NSDP) for 2014-2018, is to reduce poverty while fostering economic growth at a steady rate of 7-8% per year<sup class=\"footnote-ref\"><a href=\"#fn2\" id=\"fnref2\">[2]</a></sup>.\n",
            "goal:\n",
            "--------------------------------------------------------------------------------\n",
            "True Goal 1\n"
          ]
        }
      ],
      "source": [
        "def in_context_prompt(df, pred_column, obs_column, num_samples=3):\n",
        "    '''\n",
        "        Args:\n",
        "            df (pd.DataFrame): the dataframe that we are sampling from\n",
        "            pred_column (str): the column that we are predicting\n",
        "            obs_column (str): the column that we are observing\n",
        "            num_samples (int): the number of in-context learning samples to add to the prompt\n",
        "        Returns:\n",
        "            prompt (str): the full prompt for the model\n",
        "    '''\n",
        "    # as before we build the instruction\n",
        "    prompt = build_instruction(pred_column, obs_column)\n",
        "    \n",
        "    # for each icl example we sample a random row from the dataframe\n",
        "    for i in range(num_samples):\n",
        "        row = df.sample(1)\n",
        "        # note that we set test_input_only to false since we want to include the goal label for the icl examples\n",
        "        prompt += build_single_prompt(row.iloc[0], pred_column, obs_column, test_input_only=False)\n",
        "    return prompt\n",
        "\n",
        "\n",
        "# We need to split df_full into train/val such that the ICL examples dont overlap with the test example\n",
        "# df_train = df_full.sample(frac=0.8, random_state=42)\n",
        "# df_val = df_full.drop(df_train.index)\n",
        "\n",
        "icl_prompt_example = in_context_prompt(df_train, pred_column='goal', obs_column='text', num_samples=5)\n",
        "\n",
        "# sample a random row from the dataframe to use as a test example\n",
        "random_test_row = df_val.sample(1).iloc[0]\n",
        "test_input = build_single_prompt(random_test_row, pred_column='goal', obs_column='text', test_input_only=True)\n",
        "\n",
        "icl_prompt_example += test_input\n",
        "icl_prompt_example += f'{\"goal\"}:'\n",
        "print(icl_prompt_example)\n",
        "print('-'*80)\n",
        "print('True Goal', random_test_row['goal'])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "HmFrd9T0AIiR"
      },
      "source": [
        "### That's it! We now have a prompt with the instruction, 5 in-context learning examples, and the input text to classify.\n",
        "- For evaluation we will calculate the accuracy of the model on the test set.\n",
        "- Since each sentence can have **multiple** SDG goals labeled, we will calculate whether the model can predict **any** of the correct labels.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 73,
      "metadata": {},
      "outputs": [],
      "source": [
        "def calculate_accuracy(true_, pred_):\n",
        "    ''' Calculate accuracy of predictions\n",
        "        There could be multiple true labels for each example\n",
        "        There is only one predicted label for each example\n",
        "        Args:\n",
        "            true_ (list): list of lists of true labels\n",
        "            pred_ (list): list of predicted labels\n",
        "        Returns:\n",
        "            accuracy (float): the accuracy of the predictions\n",
        "    '''\n",
        "    assert len(true_) == len(pred_)\n",
        "    # accuracy = sum([t==p for t, p in zip(true_, pred_)])/len(true_) # this is when there is only one true label\n",
        "    right = 0\n",
        "    for t, p in zip(true_, pred_):\n",
        "        assert type(t) == list\n",
        "        if p in t:\n",
        "            right += 1\n",
        "    return right/len(true_)\n",
        "\n",
        "def most_common(lst):\n",
        "    ''' helper function to select the most common element in a list\n",
        "        Args:\n",
        "            lst: list of elements\n",
        "        Returns:\n",
        "            most common element in the list\n",
        "    '''\n",
        "    # print('lst', lst)\n",
        "    # print('type(lst)', type(lst)\n",
        "    if type(lst) == str:\n",
        "        return lst\n",
        "    return max(set(lst), key=lst.count)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Finally we are going to put this all together into a run_eval function\n",
        "- This is the function that we will pass different arguments to in order to run different experiments."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 74,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AVTCdFUxAIiR",
        "outputId": "20d134e2-696e-4534-e747-c2043cccac1a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "accuracy 0.0\n",
            "\n"
          ]
        }
      ],
      "source": [
        "def run_eval(df_train, df_test, pred_column, obs_column, num_samples, num_icl, pay=False):\n",
        "    '''\n",
        "        Args:\n",
        "            df_train (pd.DataFrame): the dataframe that we are sampling from for ICL examples\n",
        "            df_test (pd.DataFrame): the dataframe that we are sampling from for test examples\n",
        "            pred_column (str): the column name that we are predicting\n",
        "            obs_column (str): the column name that we are observing\n",
        "            num_samples (int): the number of test examples to evaluate on\n",
        "            num_icl (int): the number of in-context learning examples to include in the prompt\n",
        "            pay (bool): If False, return a dummy response. If True, send the prompt to the model API\n",
        "        Returns:\n",
        "        true_ (list): list of lists of true labels\n",
        "        pred_ (list): list of predicted labels\n",
        "        test_texts (list): list of the test examples\n",
        "        test_prompt (str): the full last prompt used for the model for debugging/sanity check purposes\n",
        "        test_ids (list): list of the ids of the test examples (this is useful such that the same test examples are not labeled multiple times)\n",
        "\n",
        "\n",
        "    '''    \n",
        "    # Note if we set the test_prompt outside the for loop here \n",
        "    # Then are using the same ICL prompt for all test examples.\n",
        "    # If we do so, we can then more easily get batch predictions from the API\n",
        "    # However, if we sample unlucky ICL examples, then the model may not perform as well.\n",
        "    # So we will leave it inside the for loop for now.\n",
        "    # test_prompt = in_context_prompt(df_train, pred_column, obs_column, num_icl)\n",
        "    ################################################################################\n",
        "\n",
        "    # Recall that one piece of text has many goals and targets. \n",
        "    # So in our dataframe there multiple rows which have the same text.\n",
        "    # This line to group makes sure that only unique texts are present.\n",
        "    # It also aggregates the multiple goals for the same text into a list.\n",
        "    df_group = df_test.groupby(obs_column, as_index=False).agg(lambda x: list(x))\n",
        "    \n",
        "    ######## Another strategy would be to use majority SDG goal\n",
        "    # assert df_group[pred_column].apply(lambda x: type(x) == list).all()\n",
        "    # df_group['majority'] = df_group[pred_column].apply(lambda x: most_common(x))\n",
        "    ################################################################################\n",
        "   \n",
        "\n",
        "    # make sure that the text column is a list and only has one unique element\n",
        "    if obs_column != 'text':\n",
        "        assert df_group['text'].apply(lambda x: len(set(x)) == 1).all()\n",
        "        df_group['text'] = df_group['text'].apply(lambda x: x[0])\n",
        "        assert len(set(df_group['text'])) == 1\n",
        "\n",
        "    # Sample num_samples test examples\n",
        "    test_rows = df_group.sample(num_samples)\n",
        "    \n",
        "    # Get the test_ids\n",
        "    test_ids = test_rows['__index_level_0__'].tolist()\n",
        "\n",
        "    true_ = []\n",
        "    pred_ = []\n",
        "    test_texts = []\n",
        "    # For each test example, get the model prediction\n",
        "    for i in range(num_samples):\n",
        "        test_row = test_rows.iloc[i]\n",
        "        text = test_row['text']\n",
        "        true = test_row[pred_column]\n",
        "        assert type(true) == list\n",
        "\n",
        "        test_prompt = in_context_prompt(df_train, pred_column, obs_column, num_icl)\n",
        "        test_input = build_single_prompt(test_row, pred_column, obs_column, test_input_only=True)\n",
        "\n",
        "        test_prompt += test_input\n",
        "        test_prompt += f'{pred_column}:'\n",
        "\n",
        "        test_texts.append(text)\n",
        "        true_.append(true)\n",
        "        \n",
        "        response = get_model_response(test_prompt, pay)\n",
        "        pred = response['choices'][0]['text'] #this is hardcoded for openai model\n",
        "        pred_.append(pred)\n",
        "    \n",
        "    return true_, pred_, test_texts, test_prompt, test_ids\n",
        "\n",
        "true_, pred_, test_texts, test_prompt, test_ids = run_eval(df_train, df_val, pred_column='goal', obs_column='text', num_samples=10, num_icl=5, pay=False)# switch pay to True to send the prompt to the OpenAI API\n",
        "\n",
        "accuracy = calculate_accuracy(true_, pred_)\n",
        "print('accuracy', accuracy)\n",
        "print()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Checking Models Prediction\n",
        "- Recall that since we are using prompting we are interpreting the model's textual output as a label.\n",
        "- We are mapping the model's output of \"3\" to the label which corresponds to SDG Goal \"3\".\n",
        "- But what if the model outputs a label outside of the range of our labels? or even fails to output a number but instead outputs a different word?\n",
        "- We can check for these cases and see how often they occur."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 75,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "count_not_digit 0\n",
            "count_in_range 0\n"
          ]
        }
      ],
      "source": [
        "def check_predictions(pred_):\n",
        "    ''' - Sanity check to make sure that all predictions are integers\n",
        "        - and also are indexed in between the number of classes\n",
        "        Args:\n",
        "            pred_ (list): list of predicted labels\n",
        "        Returns:\n",
        "            None\n",
        "    '''\n",
        "    count_not_digit = 0\n",
        "    count_in_range = 0\n",
        "    for p in pred_:\n",
        "        if not p.isdigit():\n",
        "            count_not_digit += 1\n",
        "        elif int(p) not in list(range(1, 1+N_PRED_CLASSES)):\n",
        "            count_in_range += 1\n",
        "        # assert p.isdigit()\n",
        "        # assert int(p) in list(range(1, 1+N_PRED_CLASSES))\n",
        "    print('count_not_digit', count_not_digit)\n",
        "    print('count_in_range', count_in_range)\n",
        "    \n",
        "check_predictions(pred_)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The large models from OpenAI have been tuned specifically to follow complex prompts or instructions. This is one reason why there should be only a few cases where the model fails to output a number or outputs a number outside of the range of our labels."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 76,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "avg random acc: 0.0583\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAGdCAYAAAA44ojeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAhvklEQVR4nO3de3CU5d2H8W8OJBx3Q4DsEg2neghBkAo1bCueiAQaFYcwRctgdBhsmcBUIhbTIih2mgwyQnU4dKgKtmIqHQ8VFMEoOIXlYIQ2gjDAhAlO2ARlyAYsGyD3+8c72XYlqJvsJvem12dmp2Sfe5/cvz6uudxsQpwxxggAAMAi8R29AQAAgG8iUAAAgHUIFAAAYB0CBQAAWIdAAQAA1iFQAACAdQgUAABgHQIFAABYJ7GjN9AaTU1NqqmpUa9evRQXF9fR2wEAAN+DMUYNDQ1KT09XfPy3v0YSk4FSU1OjjIyMjt4GAABohRMnTujqq6/+1jUxGSi9evWS9P8DOhyODt4NAAD4Pvx+vzIyMoJfx79NTAZK87d1HA4HgQIAQIz5Pm/P4E2yAADAOgQKAACwDoECAACsQ6AAAADrECgAAMA6BAoAALAOgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwTmJHb8BGg57YFJXzHi/Ni8p5AQDobHgFBQAAWIdAAQAA1iFQAACAdQgUAABgHQIFAABYh0ABAADWIVAAAIB1CBQAAGAdAgUAAFiHQAEAANYhUAAAgHUIFAAAYB0CBQAAWIdAAQAA1iFQAACAdQgUAABgHQIFAABYh0ABAADWIVAAAIB1CBQAAGAdAgUAAFiHQAEAANYhUAAAgHUIFAAAYB0CBQAAWIdAAQAA1iFQAACAdQgUAABgHQIFAABYh0ABAADWIVAAAIB1CBQAAGAdAgUAAFiHQAEAANYhUAAAgHUIFAAAYJ2wAuWpp55SXFxcyC0zMzN4/Pz58yosLFSfPn3Us2dP5efnq7a2NuQc1dXVysvLU/fu3ZWWlqbHH39cFy9ejMw0AACgU0gM9wHDhg3TBx988J8TJP7nFHPnztWmTZu0YcMGOZ1OzZ49W5MnT9aOHTskSZcuXVJeXp7cbrd27typkydP6sEHH1SXLl30+9//PgLjAACAziDsQElMTJTb7b7s/vr6er344otav3697rzzTknSyy+/rKFDh2rXrl0aM2aMtmzZooMHD+qDDz6Qy+XSyJEj9cwzz2j+/Pl66qmnlJSU1PaJAABAzAv7PShHjhxRenq6hgwZomnTpqm6ulqSVFFRoQsXLignJye4NjMzUwMGDJDX65Ukeb1eDR8+XC6XK7gmNzdXfr9fBw4cuOLnDAQC8vv9ITcAANB5hRUo2dnZWrt2rTZv3qxVq1apqqpKY8eOVUNDg3w+n5KSkpSSkhLyGJfLJZ/PJ0ny+XwhcdJ8vPnYlZSUlMjpdAZvGRkZ4WwbAADEmLC+xTNx4sTgn0eMGKHs7GwNHDhQr7/+urp16xbxzTUrLi5WUVFR8GO/30+kAADQibXpx4xTUlJ03XXX6ejRo3K73WpsbNSZM2dC1tTW1gbfs+J2uy/7qZ7mj1t6X0uz5ORkORyOkBsAAOi82hQoZ8+e1bFjx9S/f3+NGjVKXbp0UXl5efD44cOHVV1dLY/HI0nyeDyqrKxUXV1dcM3WrVvlcDiUlZXVlq0AAIBOJKxv8cybN0/33HOPBg4cqJqaGi1atEgJCQl64IEH5HQ6NWPGDBUVFSk1NVUOh0Nz5syRx+PRmDFjJEnjx49XVlaWpk+friVLlsjn82nBggUqLCxUcnJyVAYEAACxJ6xA+eKLL/TAAw/oq6++Ur9+/XTLLbdo165d6tevnyRp2bJlio+PV35+vgKBgHJzc7Vy5crg4xMSErRx40bNmjVLHo9HPXr0UEFBgRYvXhzZqQAAQEyLM8aYjt5EuPx+v5xOp+rr66PyfpRBT2yK+Dkl6XhpXlTOCwBALAjn6zd/Fw8AALAOgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwDoECAACsQ6AAAADrECgAAMA6BAoAALAOgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwDoECAACsQ6AAAADrECgAAMA6BAoAALAOgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwDoECAACsQ6AAAADrECgAAMA6BAoAALAOgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwDoECAACsQ6AAAADrECgAAMA6BAoAALAOgQIAAKxDoAAAAOu0KVBKS0sVFxenRx99NHjf+fPnVVhYqD59+qhnz57Kz89XbW1tyOOqq6uVl5en7t27Ky0tTY8//rguXrzYlq0AAIBOpNWBsnfvXv3xj3/UiBEjQu6fO3eu3nnnHW3YsEHbt29XTU2NJk+eHDx+6dIl5eXlqbGxUTt37tS6deu0du1aLVy4sPVTAACATqVVgXL27FlNmzZNa9asUe/evYP319fX68UXX9Rzzz2nO++8U6NGjdLLL7+snTt3ateuXZKkLVu26ODBg/rLX/6ikSNHauLEiXrmmWe0YsUKNTY2RmYqAAAQ01oVKIWFhcrLy1NOTk7I/RUVFbpw4ULI/ZmZmRowYIC8Xq8kyev1avjw4XK5XME1ubm58vv9OnDgQGu2AwAAOpnEcB9QVlamTz/9VHv37r3smM/nU1JSklJSUkLud7lc8vl8wTX/HSfNx5uPtSQQCCgQCAQ/9vv94W4bAADEkLBeQTlx4oR+9atf6dVXX1XXrl2jtafLlJSUyOl0Bm8ZGRnt9rkBAED7CytQKioqVFdXp5tuukmJiYlKTEzU9u3b9fzzzysxMVEul0uNjY06c+ZMyONqa2vldrslSW63+7Kf6mn+uHnNNxUXF6u+vj54O3HiRDjbBgAAMSasQBk3bpwqKyu1f//+4G306NGaNm1a8M9dunRReXl58DGHDx9WdXW1PB6PJMnj8aiyslJ1dXXBNVu3bpXD4VBWVlaLnzc5OVkOhyPkBgAAOq+w3oPSq1cv3XDDDSH39ejRQ3369AneP2PGDBUVFSk1NVUOh0Nz5syRx+PRmDFjJEnjx49XVlaWpk+friVLlsjn82nBggUqLCxUcnJyhMYCAACxLOw3yX6XZcuWKT4+Xvn5+QoEAsrNzdXKlSuDxxMSErRx40bNmjVLHo9HPXr0UEFBgRYvXhzprQAAgBgVZ4wxHb2JcPn9fjmdTtXX10fl2z2DntgU8XNK0vHSvKicFwCAWBDO12/+Lh4AAGAdAgUAAFiHQAEAANYhUAAAgHUIFAAAYB0CBQAAWIdAAQAA1iFQAACAdQgUAABgHQIFAABYh0ABAADWIVAAAIB1CBQAAGAdAgUAAFiHQAEAANYhUAAAgHUIFAAAYB0CBQAAWIdAAQAA1iFQAACAdQgUAABgHQIFAABYh0ABAADWIVAAAIB1CBQAAGAdAgUAAFiHQAEAANYhUAAAgHUIFAAAYB0CBQAAWIdAAQAA1iFQAACAdQgUAABgHQIFAABYh0ABAADWIVAAAIB1CBQAAGAdAgUAAFiHQAEAANYhUAAAgHUIFAAAYB0CBQAAWIdAAQAA1iFQAACAdQgUAABgHQIFAABYh0ABAADWIVAAAIB1CBQAAGAdAgUAAFiHQAEAANYhUAAAgHUSw1m8atUqrVq1SsePH5ckDRs2TAsXLtTEiRMlSefPn9djjz2msrIyBQIB5ebmauXKlXK5XMFzVFdXa9asWfroo4/Us2dPFRQUqKSkRImJYW0FuKJBT2yK2rmPl+ZF7dwAgP8I6xWUq6++WqWlpaqoqNAnn3yiO++8U5MmTdKBAwckSXPnztU777yjDRs2aPv27aqpqdHkyZODj7906ZLy8vLU2NionTt3at26dVq7dq0WLlwY2akAAEBMizPGmLacIDU1Vc8++6ymTJmifv36af369ZoyZYok6dChQxo6dKi8Xq/GjBmj9957T3fffbdqamqCr6qsXr1a8+fP16lTp5SUlPS9Pqff75fT6VR9fb0cDkdbtt+iaP0XOP/13T54BQUA7BTO1+9Wvwfl0qVLKisr07lz5+TxeFRRUaELFy4oJycnuCYzM1MDBgyQ1+uVJHm9Xg0fPjzkWz65ubny+/3BV2FaEggE5Pf7Q24AAKDzCjtQKisr1bNnTyUnJ+uXv/yl3nzzTWVlZcnn8ykpKUkpKSkh610ul3w+nyTJ5/OFxEnz8eZjV1JSUiKn0xm8ZWRkhLttAAAQQ8IOlOuvv1779+/X7t27NWvWLBUUFOjgwYPR2FtQcXGx6uvrg7cTJ05E9fMBAICOFfaPziQlJemaa66RJI0aNUp79+7VH/7wB02dOlWNjY06c+ZMyKsotbW1crvdkiS32609e/aEnK+2tjZ47EqSk5OVnJwc7lYBAECMavPvQWlqalIgENCoUaPUpUsXlZeXB48dPnxY1dXV8ng8kiSPx6PKykrV1dUF12zdulUOh0NZWVlt3QoAAOgkwnoFpbi4WBMnTtSAAQPU0NCg9evXa9u2bXr//ffldDo1Y8YMFRUVKTU1VQ6HQ3PmzJHH49GYMWMkSePHj1dWVpamT5+uJUuWyOfzacGCBSosLOQVEgAAEBRWoNTV1enBBx/UyZMn5XQ6NWLECL3//vu66667JEnLli1TfHy88vPzQ35RW7OEhARt3LhRs2bNksfjUY8ePVRQUKDFixdHdioAABDT2vx7UDoCvwcF34bfgwIAdmqX34MCAAAQLQQKAACwDoECAACsQ6AAAADrECgAAMA6BAoAALAOgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwDoECAACsQ6AAAADrECgAAMA6BAoAALAOgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwDoECAACsQ6AAAADrECgAAMA6BAoAALAOgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwDoECAACsQ6AAAADrECgAAMA6BAoAALAOgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwDoECAACsE1aglJSU6Ec/+pF69eqltLQ03XfffTp8+HDImvPnz6uwsFB9+vRRz549lZ+fr9ra2pA11dXVysvLU/fu3ZWWlqbHH39cFy9ebPs0AACgUwgrULZv367CwkLt2rVLW7du1YULFzR+/HidO3cuuGbu3Ll65513tGHDBm3fvl01NTWaPHly8PilS5eUl5enxsZG7dy5U+vWrdPatWu1cOHCyE0FAABiWpwxxrT2wadOnVJaWpq2b9+uW2+9VfX19erXr5/Wr1+vKVOmSJIOHTqkoUOHyuv1asyYMXrvvfd09913q6amRi6XS5K0evVqzZ8/X6dOnVJSUtJ3fl6/3y+n06n6+no5HI7Wbv+KBj2xKeLnlKTjpXlROS9CRev6SVxDAGiLcL5+t+k9KPX19ZKk1NRUSVJFRYUuXLignJyc4JrMzEwNGDBAXq9XkuT1ejV8+PBgnEhSbm6u/H6/Dhw40OLnCQQC8vv9ITcAANB5tTpQmpqa9Oijj+onP/mJbrjhBkmSz+dTUlKSUlJSQta6XC75fL7gmv+Ok+bjzcdaUlJSIqfTGbxlZGS0dtsAACAGtDpQCgsL9dlnn6msrCyS+2lRcXGx6uvrg7cTJ05E/XMCAICOk9iaB82ePVsbN27Uxx9/rKuvvjp4v9vtVmNjo86cORPyKkptba3cbndwzZ49e0LO1/xTPs1rvik5OVnJycmt2SoAAIhBYb2CYozR7Nmz9eabb+rDDz/U4MGDQ46PGjVKXbp0UXl5efC+w4cPq7q6Wh6PR5Lk8XhUWVmpurq64JqtW7fK4XAoKyurLbMAAIBOIqxXUAoLC7V+/Xq9/fbb6tWrV/A9I06nU926dZPT6dSMGTNUVFSk1NRUORwOzZkzRx6PR2PGjJEkjR8/XllZWZo+fbqWLFkin8+nBQsWqLCwkFdJAACApDADZdWqVZKk22+/PeT+l19+WQ899JAkadmyZYqPj1d+fr4CgYByc3O1cuXK4NqEhARt3LhRs2bNksfjUY8ePVRQUKDFixe3bRIAANBphBUo3+dXpnTt2lUrVqzQihUrrrhm4MCBevfdd8P51AAA4H8IfxcPAACwDoECAACsQ6AAAADrECgAAMA6BAoAALAOgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwDoECAACsQ6AAAADrECgAAMA6BAoAALAOgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwDoECAACsQ6AAAADrECgAAMA6BAoAALAOgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwDoECAACsQ6AAAADrECgAAMA6BAoAALAOgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwTmJHbwAAvmnQE5uidu7jpXlROzeAyOEVFAAAYJ2wA+Xjjz/WPffco/T0dMXFxemtt94KOW6M0cKFC9W/f39169ZNOTk5OnLkSMia06dPa9q0aXI4HEpJSdGMGTN09uzZNg0CAAA6j7AD5dy5c7rxxhu1YsWKFo8vWbJEzz//vFavXq3du3erR48eys3N1fnz54Nrpk2bpgMHDmjr1q3auHGjPv74Yz3yyCOtnwIAAHQqYb8HZeLEiZo4cWKLx4wxWr58uRYsWKBJkyZJkl555RW5XC699dZbuv/++/X5559r8+bN2rt3r0aPHi1JeuGFF/TTn/5US5cuVXp6ehvGAQAAnUFE34NSVVUln8+nnJyc4H1Op1PZ2dnyer2SJK/Xq5SUlGCcSFJOTo7i4+O1e/fuFs8bCATk9/tDbgAAoPOKaKD4fD5JksvlCrnf5XIFj/l8PqWlpYUcT0xMVGpqanDNN5WUlMjpdAZvGRkZkdw2AACwTEz8FE9xcbHq6+uDtxMnTnT0lgAAQBRFNFDcbrckqba2NuT+2tra4DG32626urqQ4xcvXtTp06eDa74pOTlZDocj5AYAADqviAbK4MGD5Xa7VV5eHrzP7/dr9+7d8ng8kiSPx6MzZ86ooqIiuObDDz9UU1OTsrOzI7kdAAAQo8L+KZ6zZ8/q6NGjwY+rqqq0f/9+paamasCAAXr00Uf1u9/9Ttdee60GDx6sJ598Uunp6brvvvskSUOHDtWECRM0c+ZMrV69WhcuXNDs2bN1//338xM8AABAUisC5ZNPPtEdd9wR/LioqEiSVFBQoLVr1+rXv/61zp07p0ceeURnzpzRLbfcos2bN6tr167Bx7z66quaPXu2xo0bp/j4eOXn5+v555+PwDgAAKAzCDtQbr/9dhljrng8Li5Oixcv1uLFi6+4JjU1VevXrw/3UwMAgP8RMfFTPAAA4H8LgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwDoECAACsQ6AAAADrECgAAMA6BAoAALAOgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwDoECAACsQ6AAAADrECgAAMA6BAoAALAOgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwDoECAACsQ6AAAADrECgAAMA6BAoAALAOgQIAAKxDoAAAAOsQKAAAwDoECgAAsA6BAgAArEOgAAAA6xAoAADAOgQKAACwTmJHbwAA0PkMemJTVM57vDQvKueFfXgFBQAAWIdAAQAA1iFQAACAdQgUAABgHQIFAABYp0MDZcWKFRo0aJC6du2q7Oxs7dmzpyO3AwAALNFhgfLXv/5VRUVFWrRokT799FPdeOONys3NVV1dXUdtCQAAWKLDAuW5557TzJkz9fDDDysrK0urV69W9+7d9dJLL3XUlgAAgCU65Be1NTY2qqKiQsXFxcH74uPjlZOTI6/Xe9n6QCCgQCAQ/Li+vl6S5Pf7o7K/psDXUTlvtPaLUNG6fhLXsL1wDWMf/x5tHzcsej9q5/7s6dyIn7P5+hljvnNthwTKl19+qUuXLsnlcoXc73K5dOjQocvWl5SU6Omnn77s/oyMjKjtMRqcyzt6B2grrmHs4xrGNq5f+4nm/9cNDQ1yOp3fuiYmftV9cXGxioqKgh83NTXp9OnT6tOnj+Li4iL6ufx+vzIyMnTixAk5HI6IntsGzBf7OvuMzBf7OvuMnX0+KXozGmPU0NCg9PT071zbIYHSt29fJSQkqLa2NuT+2tpaud3uy9YnJycrOTk55L6UlJRoblEOh6PT/oMnMV9n0NlnZL7Y19ln7OzzSdGZ8bteOWnWIW+STUpK0qhRo1ReXh68r6mpSeXl5fJ4PB2xJQAAYJEO+xZPUVGRCgoKNHr0aN18881avny5zp07p4cffrijtgQAACzRYYEydepUnTp1SgsXLpTP59PIkSO1efPmy944296Sk5O1aNGiy76l1FkwX+zr7DMyX+zr7DN29vkkO2aMM9/nZ30AAADaEX8XDwAAsA6BAgAArEOgAAAA6xAoAADAOp0+UFasWKFBgwapa9euys7O1p49e751/YYNG5SZmamuXbtq+PDhevfdd0OOG2O0cOFC9e/fX926dVNOTo6OHDkSzRG+U6RnfOihhxQXFxdymzBhQjRH+FbhzHfgwAHl5+dr0KBBiouL0/Lly9t8zmiL9HxPPfXUZdcvMzMzihN8t3BmXLNmjcaOHavevXurd+/eysnJuWy9bc/DSM8Xy8/BN954Q6NHj1ZKSop69OihkSNH6s9//nPIGtuunxT5GWP5Gv63srIyxcXF6b777gu5v12uoenEysrKTFJSknnppZfMgQMHzMyZM01KSoqpra1tcf2OHTtMQkKCWbJkiTl48KBZsGCB6dKli6msrAyuKS0tNU6n07z11lvmn//8p7n33nvN4MGDzb///e/2GitENGYsKCgwEyZMMCdPngzeTp8+3V4jhQh3vj179ph58+aZ1157zbjdbrNs2bI2nzOaojHfokWLzLBhw0Ku36lTp6I8yZWFO+PPf/5zs2LFCrNv3z7z+eefm4ceesg4nU7zxRdfBNfY9DyMxnyx/Bz86KOPzBtvvGEOHjxojh49apYvX24SEhLM5s2bg2tsun7GRGfGWL6GzaqqqsxVV11lxo4dayZNmhRyrD2uYacOlJtvvtkUFhYGP7506ZJJT083JSUlLa7/2c9+ZvLy8kLuy87ONr/4xS+MMcY0NTUZt9ttnn322eDxM2fOmOTkZPPaa69FYYLvFukZjfn/J9Y3/2HsKOHO998GDhzY4hfwtpwz0qIx36JFi8yNN94YwV22TVv//7548aLp1auXWbdunTHGvudhpOczpvM8B5v98Ic/NAsWLDDG2Hf9jIn8jMbE/jW8ePGi+fGPf2z+9Kc/XTZLe13DTvstnsbGRlVUVCgnJyd4X3x8vHJycuT1elt8jNfrDVkvSbm5ucH1VVVV8vl8IWucTqeys7OveM5oisaMzbZt26a0tDRdf/31mjVrlr766qvID/AdWjNfR5yztaK5lyNHjig9PV1DhgzRtGnTVF1d3dbttkokZvz666914cIFpaamSrLreRiN+Zp1huegMUbl5eU6fPiwbr31Vkl2XT8pOjM2i+VruHjxYqWlpWnGjBmXHWuvaxgTf5txa3z55Ze6dOnSZb+Z1uVy6dChQy0+xufztbje5/MFjzffd6U17SkaM0rShAkTNHnyZA0ePFjHjh3Tb37zG02cOFFer1cJCQmRH+QKWjNfR5yztaK1l+zsbK1du1bXX3+9Tp48qaefflpjx47VZ599pl69erV122GJxIzz589Xenp68F+GNj0PozGfFPvPwfr6el111VUKBAJKSEjQypUrddddd0my6/pJ0ZlRiu1r+I9//EMvvvii9u/f3+Lx9rqGnTZQ0Hr3339/8M/Dhw/XiBEj9IMf/EDbtm3TuHHjOnBn+D4mTpwY/POIESOUnZ2tgQMH6vXXX2/xv4ZsVlpaqrKyMm3btk1du3bt6O1E3JXmi/XnYK9evbR//36dPXtW5eXlKioq0pAhQ3T77bd39NYi5rtmjNVr2NDQoOnTp2vNmjXq27dvh+6l036Lp2/fvkpISFBtbW3I/bW1tXK73S0+xu12f+v65v8N55zRFI0ZWzJkyBD17dtXR48ebfumw9Ca+TrinK3VXntJSUnRdddd1+7XT2rbjEuXLlVpaam2bNmiESNGBO+36XkYjflaEmvPwfj4eF1zzTUaOXKkHnvsMU2ZMkUlJSWS7Lp+UnRmbEmsXMNjx47p+PHjuueee5SYmKjExES98sor+vvf/67ExEQdO3as3a5hpw2UpKQkjRo1SuXl5cH7mpqaVF5eLo/H0+JjPB5PyHpJ2rp1a3D94MGD5Xa7Q9b4/X7t3r37iueMpmjM2JIvvvhCX331lfr37x+ZjX9PrZmvI87ZWu21l7Nnz+rYsWPtfv2k1s+4ZMkSPfPMM9q8ebNGjx4dcsym52E05mtJrD8Hm5qaFAgEJNl1/aTozNiSWLmGmZmZqqys1P79+4O3e++9V3fccYf279+vjIyM9ruGEXu7rYXKyspMcnKyWbt2rTl48KB55JFHTEpKivH5fMYYY6ZPn26eeOKJ4PodO3aYxMREs3TpUvP555+bRYsWtfhjxikpKebtt982//rXv8ykSZM6/MfjIjljQ0ODmTdvnvF6vaaqqsp88MEH5qabbjLXXnutOX/+vPXzBQIBs2/fPrNv3z7Tv39/M2/ePLNv3z5z5MiR733OWJ/vscceM9u2bTNVVVVmx44dJicnx/Tt29fU1dW1+3zGhD9jaWmpSUpKMn/7299CfkSzoaEhZI0tz8NIzxfrz8Hf//73ZsuWLebYsWPm4MGDZunSpSYxMdGsWbMmuMam62dM5GeM9Wv4TS39RFJ7XMNOHSjGGPPCCy+YAQMGmKSkJHPzzTebXbt2BY/ddtttpqCgIGT966+/bq677jqTlJRkhg0bZjZt2hRyvKmpyTz55JPG5XKZ5ORkM27cOHP48OH2GOWKIjnj119/bcaPH2/69etnunTpYgYOHGhmzpzZIV+8m4UzX1VVlZF02e2222773udsb5Geb+rUqaZ///4mKSnJXHXVVWbq1Knm6NGj7TjR5cKZceDAgS3OuGjRouAa256HkZwv1p+Dv/3tb80111xjunbtanr37m08Ho8pKysLOZ9t18+YyM4Y69fwm1oKlPa4hnHGGBO512MAAADartO+BwUAAMQuAgUAAFiHQAEAANYhUAAAgHUIFAAAYB0CBQAAWIdAAQAA1iFQAACAdQgUAABgHQIFAABYh0ABAADWIVAAAIB1/g9RoEYyprt8pQAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "def plot_random_histogram(true_):\n",
        "    ''' Plot a histogram of accuracies of a random classifier\n",
        "        Args:\n",
        "            true_ (list): list of lists of true labels\n",
        "        Returns:\n",
        "            None\n",
        "    '''\n",
        "    num_trials = 1000\n",
        "    random_accuracies = []\n",
        "    for i in range(num_trials):\n",
        "        random_pred = np.random.choice(list(range(1, 1+N_PRED_CLASSES)), size=len(true_))\n",
        "        random_pred = [str(x) for x in random_pred]\n",
        "        \n",
        "        random_accuracy = calculate_accuracy(true_, random_pred)\n",
        "        \n",
        "        random_accuracies.append(random_accuracy)\n",
        "    # histogram\n",
        "    import matplotlib.pyplot as plt\n",
        "    plt.hist(random_accuracies, bins=20)\n",
        "    avg_acc = np.mean(random_accuracies)\n",
        "    \n",
        "    # this is how a random uniform classifier would perform\n",
        "    print(f'avg random acc: {avg_acc}')\n",
        "\n",
        "plot_random_histogram(true_)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Using longer text contexts.\n",
        "- We'll recall that the SDG text has annotations that are actually from the full text National climate action plan documents or NDCs.\n",
        "- The dataset has already been preprocessed to align the text chunks that were annotated with their corresponding start and end indices in the full text documents.\n",
        "- Here we will change the prompts to include longer \"contexts\" around the \"texts\" since this increased context could help the model make better predictions.\n",
        "- Since this longer context may itself have different annotations with different SDG goals, we will also need to make the particular text we want to classify stand out.\n",
        "- For this will change the instruction prompt, the ICL examples, as well as the the test input_text as well to use a special token to mark the text.\n",
        "- To start lets load the full text documents and see how they look."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 77,
      "metadata": {},
      "outputs": [],
      "source": [
        "def load_documents_from_hub():\n",
        "    '''\n",
        "        Load the full text of the documents from the huggingface hub.\n",
        "        Returns:\n",
        "            documents (list): list of dictionaries, \n",
        "                each dictionary has the name of the file as the key and the full text as the value\n",
        "    '''\n",
        "    # Download from a dataset\n",
        "    filepath = hf_hub_download(repo_id=\"dspoka/sdg-single\", filename=\"ndc-all.json\", repo_type=\"dataset\")\n",
        "    with open(filepath) as f:\n",
        "        documents = json.load(f)\n",
        "    return documents\n",
        "\n",
        "\n",
        "documents = load_documents_from_hub()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 78,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Matched train: (8920, 13)\tval: (2507, 13)\ttest: (2101, 13)\n"
          ]
        }
      ],
      "source": [
        "def span_check(span, doc):\n",
        "    ''' Check that the span is valid with some basic assertions\n",
        "        Args:\n",
        "            span (tuple): tuple of (start, end) indices\n",
        "            doc (str): document text\n",
        "        Returns:\n",
        "            None\n",
        "    '''\n",
        "    # check that all starts and ends are within the document\n",
        "    start, end = span\n",
        "    assert start < end\n",
        "    assert start >= 0\n",
        "    assert end <= len(doc)\n",
        "    \n",
        "def prepare_matched_df(df):\n",
        "    ''' Prepare the dataframe to be used for matching.\n",
        "        Args:\n",
        "            df (pd.DataFrame): dataframe with the columns 'start' and 'end'\n",
        "        Returns:\n",
        "            df (pd.DataFrame): dataframe where the text was matched with start/end indices\n",
        "    '''\n",
        "    \n",
        "    # 1. Lets subselect only data that has been matched by checking whether the start and end indices are present\n",
        "    df = df[df['start'].notnull()]\n",
        "    df = df[df['end'].notnull()]\n",
        "    # 2. Lets also make sure start and end are integers not floats.\n",
        "    df['start'] = df['start'].astype(int)\n",
        "    df['end'] = df['end'].astype(int)\n",
        "    return df\n",
        "\n",
        "# drop rows that don't have start and end indices\n",
        "df_train = prepare_matched_df(df_train)\n",
        "df_val = prepare_matched_df(df_val)\n",
        "df_test = prepare_matched_df(df_test)\n",
        "print(f'Matched train: {df_train.shape}\\tval: {df_val.shape}\\ttest: {df_test.shape}')\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 79,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "d>Carbon sequestration on forest/rangelands, and forest carbon skills. Funding institutional capacity to monitor and verify projects. Better spatial planning for community and production agriculture. **Reduce rural peoples’ dependence on fuel for cooking and heating.**</td>\n",
            "<td>100 Million/Year</td>\n",
            "</tr>\n",
            "<tr>\n",
            "<td>Industry and Mining</td>\n",
            "<td>Cleaner coal mining, leave-it-in-the-ground approaches, combustion, and transportation of minerals. Hydrocarbon fields manag\n"
          ]
        }
      ],
      "source": [
        "def add_context(df, documents, window_size=200, special_token=\"**\"):\n",
        "    ''' For each sentence add a window of context before and after the sentence.\n",
        "        Add a special token to denote the start and end of the sentence.\n",
        "        Args:\n",
        "            df (pd.DataFrame): dataframe with the columns 'filename', 'start', and 'end'\n",
        "            documents (list): list of dictionaries,\n",
        "            window_size (int): number of characters to include before and after the sentence\n",
        "            special_token (str): special token to denote the start and end of the sentence\n",
        "        Returns:\n",
        "            df (pd.DataFrame): dataframe with the column 'context' added\n",
        "\n",
        "    '''\n",
        "    # add a new column to the dataframe called 'context'\n",
        "    df['context'] = ''\n",
        "    for i, row in df.iterrows():\n",
        "        filename = row['filename']\n",
        "        start = row['start']\n",
        "        end = row['end']\n",
        "        span = (start, end)\n",
        "        \n",
        "        doc = documents[filename]\n",
        "        span_check(span, doc)\n",
        "\n",
        "        start_context = max(0, start-window_size)\n",
        "        end_context = min(len(doc), end+window_size)\n",
        "        # print(f'start_context: {start_context}\\tend_context: {end_context}')\n",
        "        context = doc[start_context:start] + special_token + doc[start:end] + special_token + doc[end:end_context]\n",
        "        df.at[i, 'context'] = context\n",
        "    return df\n",
        "\n",
        "\n",
        "WINDOW_SIZE = 200\n",
        "df_train = add_context(df_train, documents, window_size=WINDOW_SIZE, special_token=SPECIAL_TOKEN)\n",
        "df_val = add_context(df_val, documents, window_size=WINDOW_SIZE, special_token=SPECIAL_TOKEN)\n",
        "print(df_train['context'].iloc[0])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### We are ready to run evaluation again on the longer contexts\n",
        "- before doing that we are going to include a few house keeping functions to save what we are evaluating and the results of the evaluation\n",
        "- We will also use an eval_dict to keep track of the parameters of our evaluation experiments"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 80,
      "metadata": {},
      "outputs": [],
      "source": [
        "OUTPUT_DIR = 'outputs/'\n",
        "if not os.path.exists(OUTPUT_DIR):\n",
        "    os.makedirs(OUTPUT_DIR)\n",
        "\n",
        "def save_json(filename, data):\n",
        "    with open(os.path.join(OUTPUT_DIR, filename), 'w') as f:\n",
        "        json.dump(data, f)\n",
        "\n",
        "def load_json(filename):\n",
        "    with open(os.path.join(OUTPUT_DIR, filename), 'r') as f:\n",
        "        data = json.load(f)\n",
        "    return data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 81,
      "metadata": {},
      "outputs": [],
      "source": [
        "# You can change the experiment here\n",
        "# If you set pay to True this will use the API\n",
        "# rough cost is 3-5 dollars for ~100 examples. (it depends on the length of the examples)\n",
        "eval_dict = {'num_icl_examples': 5,\n",
        "             'num_eval_samples': 10,\n",
        "            # 'obs_column': 'context',\n",
        "            'obs_column': 'text',\n",
        "            'pred_column': 'goal',\n",
        "            'model_name': MODEL_NAME,\n",
        "            'window_size': WINDOW_SIZE,\n",
        "            'special_token': SPECIAL_TOKEN,\n",
        "            'input_marker': INPUT_MARKER,\n",
        "            'pay': False}\n",
        "\n",
        "# just for shorthand\n",
        "e = eval_dict"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 82,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "In-Context Learning Prompt:\n",
            " Given the following Input Text predict the Sustainable Development Goal (goal) out of the following 17 options:\n",
            "Sustainable Development Goal\n",
            "1: End poverty in all its forms everywhere\n",
            "2: End hunger, achieve food security and improved nutrition and promote sustainable agriculture\n",
            "3: Ensure healthy lives and promote well-being for all at all ages\n",
            "4: Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all\n",
            "5: Achieve gender equality and empower all women and girls\n",
            "6: Ensure availability and sustainable management of water and sanitation for all\n",
            "7: Ensure access to affordable, reliable, sustainable and modern energy for all\n",
            "8: Promote sustained, inclusive and sustainable economic growth, full and productive employment and decent work for all\n",
            "9: Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation\n",
            "10: Reduce inequality within and among countries\n",
            "11: Make cities and human settlements inclusive, safe, resilient and sustainable\n",
            "12: Ensure sustainable consumption and production patterns\n",
            "13: Take urgent action to combat climate change and its impacts\n",
            "14: Conserve and sustainably use the oceans, seas and marine resources for sustainable development\n",
            "15: Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss\n",
            "16: Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels\n",
            "17: Strengthen the means of implementation and revitalize the global partnership for sustainable development\n",
            "\n",
            "\n",
            "Input Text: Establishment of model cattle farms with silvopastoral systems for demonstrations of environmentally friendly farms.\n",
            "goal:2\n",
            "\n",
            "\n",
            "Input Text: nature-based solutions play an important role to solve global challenges such as biodiversity loss and ecosystems degradation, poverty, hunger, health, water scarcity and drought, gender inequality, disaster risk reduction and climate change\n",
            "goal:15\n",
            "\n",
            "\n",
            "Input Text: Increasing public awareness to water related issue\n",
            "goal:12\n",
            "\n",
            "\n",
            "Input Text: Reduction of loss in transmission and distribution networks\n",
            "goal:7\n",
            "\n",
            "\n",
            "Input Text: Implementation of control, monitoring, and tracking systems for the appropriate use of areas of forest life\n",
            "goal:15\n",
            "\n",
            "\n",
            "Input Text: These associations would induce the transfer of technologies and inputs to farmers, not to mention social actions, in a spirit of respect for the soil and the biological richness of the surrounding environment (creation of micro-biological reserves).\n",
            "goal:\n",
            "--------------------------------------------------------------------------------\n",
            "accuracy 0.0\n",
            "--------------------------------------------------------------------------------\n"
          ]
        }
      ],
      "source": [
        "true_, pred_, test_text, test_prompt, test_ids = run_eval(df_train, df_val, e['pred_column'], \n",
        "                                            e['obs_column'], e['num_eval_samples'], \n",
        "                                            e['num_icl_examples'], e['pay'])\n",
        "\n",
        "results = {'true': true_, 'pred': pred_, 'test_text': test_text, 'test_prompt': test_prompt, 'test_ids': test_ids}\n",
        "filename = f\"{e['obs_column']}_{e['pred_column']}_icl@{e['num_icl_examples']}\"\n",
        "\n",
        "# make sure to save the results\n",
        "save_json(f'{filename}_results.json', results)\n",
        "# load the results from file\n",
        "results = load_json(f'{filename}_results.json')\n",
        "# unpack the results\n",
        "true_, pred_, test_text, test_prompt, test_ids = results['true'], results['pred'], results['test_text'], results['test_prompt'], results['test_ids']\n",
        "\n",
        "\n",
        "print(f'In-Context Learning Prompt:\\n {test_prompt}')\n",
        "print('-'*80)\n",
        "\n",
        "accuracy = calculate_accuracy(true_, pred_)\n",
        "eval_dict['accuracy'] = accuracy\n",
        "print('accuracy', accuracy)\n",
        "\n",
        "print('-'*80)\n",
        "# save the experiment ran with the accuracy\n",
        "save_json(f'{filename}_eval_dict.json', eval_dict)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "tutorial-IHiCU4EY-py3.8",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.0"
    },
    "orig_nbformat": 4,
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0a8d3c8e28964a2b83dbf16902bf35a3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "30058e6b100a4837b21bee31c619f7c7": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5b515b9cf3be4941b350a11e24aa8c26": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ab879f6b5c0e48e6a88fcef30c4a0dc5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_b60848d12d9d4743b648ebaf43122d87",
              "IPY_MODEL_bdf310baec9e4bb292a616a1c9941b44",
              "IPY_MODEL_cc76fa30bf2745899ebb1a4fdc108139"
            ],
            "layout": "IPY_MODEL_af3ccb7b556948449f317600d1361158"
          }
        },
        "af3ccb7b556948449f317600d1361158": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b60848d12d9d4743b648ebaf43122d87": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c866b6073eb54001aeede1a87f4c79a2",
            "placeholder": "​",
            "style": "IPY_MODEL_5b515b9cf3be4941b350a11e24aa8c26",
            "value": "100%"
          }
        },
        "bdf310baec9e4bb292a616a1c9941b44": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_daad8401c7b54fb98b488587d455d7e4",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_e267c4404b5145fb9465d5a89e86da1b",
            "value": 1
          }
        },
        "c866b6073eb54001aeede1a87f4c79a2": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cc76fa30bf2745899ebb1a4fdc108139": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_30058e6b100a4837b21bee31c619f7c7",
            "placeholder": "​",
            "style": "IPY_MODEL_0a8d3c8e28964a2b83dbf16902bf35a3",
            "value": " 1/1 [00:00&lt;00:00, 41.42it/s]"
          }
        },
        "daad8401c7b54fb98b488587d455d7e4": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e267c4404b5145fb9465d5a89e86da1b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}