my_DSPy_tutorial.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "mount_file_id": "1Gha8sitD--BnMksTELozeOSlq0CeyYHx",
      "authorship_tag": "ABX9TyPBq6Ic+yFs0MkMXJmXhvj0",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/edgarbc/LLM_optimizer/blob/main/my_DSPy_tutorial.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "My DSPy tutorial\n",
        "\n",
        "Example of how to use DSPy for systematic optimization of a simple LLM application.\n",
        "\n",
        "Adapted from [this blog post](https://learnbybuilding.ai/tutorials/a-gentle-introduction-to-dspy).\n",
        "\n",
        "Edgar Bermudez\n",
        "\n",
        "May 2024."
      ],
      "metadata": {
        "id": "Rb4zvOCxgfBp"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 349
        },
        "id": "cTBmzt9KeMBc",
        "outputId": "9f6ede01-4e87-4d90-bafd-6f0b56372430"
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "ModuleNotFoundError",
          "evalue": "No module named 'dotenv'",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-1-88542ea5c1fd>\u001b[0m in \u001b[0;36m<cell line: 2>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# load the credentials and secrets variables\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mdotenv\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mload_dotenv\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mload_dotenv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'dotenv'",
            "",
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0;32m\nNOTE: If your import is failing due to a missing package, you can\nmanually install dependencies using either !pip or !apt.\n\nTo view examples of installing some common dependencies, click the\n\"Open Examples\" button below.\n\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n"
          ],
          "errorDetails": {
            "actions": [
              {
                "action": "open_url",
                "actionText": "Open Examples",
                "url": "/notebooks/snippets/importing_libraries.ipynb"
              }
            ]
          }
        }
      ],
      "source": [
        "# load the credentials and secrets variables\n",
        "from dotenv import load_dotenv\n",
        "load_dotenv()"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install dspy"
      ],
      "metadata": {
        "id": "0bYEA27njKsS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install kaggle"
      ],
      "metadata": {
        "id": "1MZQnxaalpDV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Web page information\n",
        "\n",
        "For this example we are going to extract about the current Kaggle competitions. Kaggle is a learning-by-doing platform in which challenges are posted as a competition. Users submit ML solutions that are evaluated and ranked. At the end of the competitions the top-leaderboard winners get a money prize (some money-free prizes are available). In general, it is a great place to learn from the community."
      ],
      "metadata": {
        "id": "zV1gKG3nh8wp"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "l-PJRvxZtf-P"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "8sKT-0P0tgzM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now you need to get the kaggle api key from your kaggle profile to be able to get kaggle data."
      ],
      "metadata": {
        "id": "IugbwCdathk5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import files\n",
        "\n",
        "uploaded = files.upload()\n",
        "\n",
        "for fn in uploaded.keys():\n",
        "  print('User uploaded file \"{name}\" with length {length} bytes'.format(\n",
        "      name=fn, length=len(uploaded[fn])))\n",
        "\n",
        "# Then move kaggle.json into the folder where the API expects to find it.\n",
        "!mkdir -p ~/.kaggle/ && mv kaggle.json ~/.kaggle/ && chmod 600 ~/.kaggle/kaggle.json"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        },
        "id": "eH421eS_s11F",
        "outputId": "663746a0-4c26-4cbd-a93f-0b6edcdd0b4a"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-f236a172-802f-417c-9b20-8eb913923007\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-f236a172-802f-417c-9b20-8eb913923007\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script>// Copyright 2017 Google LLC\n",
              "//\n",
              "// Licensed under the Apache License, Version 2.0 (the \"License\");\n",
              "// you may not use this file except in compliance with the License.\n",
              "// You may obtain a copy of the License at\n",
              "//\n",
              "//      http://www.apache.org/licenses/LICENSE-2.0\n",
              "//\n",
              "// Unless required by applicable law or agreed to in writing, software\n",
              "// distributed under the License is distributed on an \"AS IS\" BASIS,\n",
              "// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
              "// See the License for the specific language governing permissions and\n",
              "// limitations under the License.\n",
              "\n",
              "/**\n",
              " * @fileoverview Helpers for google.colab Python module.\n",
              " */\n",
              "(function(scope) {\n",
              "function span(text, styleAttributes = {}) {\n",
              "  const element = document.createElement('span');\n",
              "  element.textContent = text;\n",
              "  for (const key of Object.keys(styleAttributes)) {\n",
              "    element.style[key] = styleAttributes[key];\n",
              "  }\n",
              "  return element;\n",
              "}\n",
              "\n",
              "// Max number of bytes which will be uploaded at a time.\n",
              "const MAX_PAYLOAD_SIZE = 100 * 1024;\n",
              "\n",
              "function _uploadFiles(inputId, outputId) {\n",
              "  const steps = uploadFilesStep(inputId, outputId);\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  // Cache steps on the outputElement to make it available for the next call\n",
              "  // to uploadFilesContinue from Python.\n",
              "  outputElement.steps = steps;\n",
              "\n",
              "  return _uploadFilesContinue(outputId);\n",
              "}\n",
              "\n",
              "// This is roughly an async generator (not supported in the browser yet),\n",
              "// where there are multiple asynchronous steps and the Python side is going\n",
              "// to poll for completion of each step.\n",
              "// This uses a Promise to block the python side on completion of each step,\n",
              "// then passes the result of the previous step as the input to the next step.\n",
              "function _uploadFilesContinue(outputId) {\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  const steps = outputElement.steps;\n",
              "\n",
              "  const next = steps.next(outputElement.lastPromiseValue);\n",
              "  return Promise.resolve(next.value.promise).then((value) => {\n",
              "    // Cache the last promise value to make it available to the next\n",
              "    // step of the generator.\n",
              "    outputElement.lastPromiseValue = value;\n",
              "    return next.value.response;\n",
              "  });\n",
              "}\n",
              "\n",
              "/**\n",
              " * Generator function which is called between each async step of the upload\n",
              " * process.\n",
              " * @param {string} inputId Element ID of the input file picker element.\n",
              " * @param {string} outputId Element ID of the output display.\n",
              " * @return {!Iterable<!Object>} Iterable of next steps.\n",
              " */\n",
              "function* uploadFilesStep(inputId, outputId) {\n",
              "  const inputElement = document.getElementById(inputId);\n",
              "  inputElement.disabled = false;\n",
              "\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  outputElement.innerHTML = '';\n",
              "\n",
              "  const pickedPromise = new Promise((resolve) => {\n",
              "    inputElement.addEventListener('change', (e) => {\n",
              "      resolve(e.target.files);\n",
              "    });\n",
              "  });\n",
              "\n",
              "  const cancel = document.createElement('button');\n",
              "  inputElement.parentElement.appendChild(cancel);\n",
              "  cancel.textContent = 'Cancel upload';\n",
              "  const cancelPromise = new Promise((resolve) => {\n",
              "    cancel.onclick = () => {\n",
              "      resolve(null);\n",
              "    };\n",
              "  });\n",
              "\n",
              "  // Wait for the user to pick the files.\n",
              "  const files = yield {\n",
              "    promise: Promise.race([pickedPromise, cancelPromise]),\n",
              "    response: {\n",
              "      action: 'starting',\n",
              "    }\n",
              "  };\n",
              "\n",
              "  cancel.remove();\n",
              "\n",
              "  // Disable the input element since further picks are not allowed.\n",
              "  inputElement.disabled = true;\n",
              "\n",
              "  if (!files) {\n",
              "    return {\n",
              "      response: {\n",
              "        action: 'complete',\n",
              "      }\n",
              "    };\n",
              "  }\n",
              "\n",
              "  for (const file of files) {\n",
              "    const li = document.createElement('li');\n",
              "    li.append(span(file.name, {fontWeight: 'bold'}));\n",
              "    li.append(span(\n",
              "        `(${file.type || 'n/a'}) - ${file.size} bytes, ` +\n",
              "        `last modified: ${\n",
              "            file.lastModifiedDate ? file.lastModifiedDate.toLocaleDateString() :\n",
              "                                    'n/a'} - `));\n",
              "    const percent = span('0% done');\n",
              "    li.appendChild(percent);\n",
              "\n",
              "    outputElement.appendChild(li);\n",
              "\n",
              "    const fileDataPromise = new Promise((resolve) => {\n",
              "      const reader = new FileReader();\n",
              "      reader.onload = (e) => {\n",
              "        resolve(e.target.result);\n",
              "      };\n",
              "      reader.readAsArrayBuffer(file);\n",
              "    });\n",
              "    // Wait for the data to be ready.\n",
              "    let fileData = yield {\n",
              "      promise: fileDataPromise,\n",
              "      response: {\n",
              "        action: 'continue',\n",
              "      }\n",
              "    };\n",
              "\n",
              "    // Use a chunked sending to avoid message size limits. See b/62115660.\n",
              "    let position = 0;\n",
              "    do {\n",
              "      const length = Math.min(fileData.byteLength - position, MAX_PAYLOAD_SIZE);\n",
              "      const chunk = new Uint8Array(fileData, position, length);\n",
              "      position += length;\n",
              "\n",
              "      const base64 = btoa(String.fromCharCode.apply(null, chunk));\n",
              "      yield {\n",
              "        response: {\n",
              "          action: 'append',\n",
              "          file: file.name,\n",
              "          data: base64,\n",
              "        },\n",
              "      };\n",
              "\n",
              "      let percentDone = fileData.byteLength === 0 ?\n",
              "          100 :\n",
              "          Math.round((position / fileData.byteLength) * 100);\n",
              "      percent.textContent = `${percentDone}% done`;\n",
              "\n",
              "    } while (position < fileData.byteLength);\n",
              "  }\n",
              "\n",
              "  // All done.\n",
              "  yield {\n",
              "    response: {\n",
              "      action: 'complete',\n",
              "    }\n",
              "  };\n",
              "}\n",
              "\n",
              "scope.google = scope.google || {};\n",
              "scope.google.colab = scope.google.colab || {};\n",
              "scope.google.colab._files = {\n",
              "  _uploadFiles,\n",
              "  _uploadFilesContinue,\n",
              "};\n",
              "})(self);\n",
              "</script> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving kaggle.json to kaggle.json\n",
            "User uploaded file \"kaggle.json\" with length 63 bytes\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import kaggle"
      ],
      "metadata": {
        "id": "B8NCWw2xmsbY"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!kaggle competitions list"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "da0ImOBpuhsN",
        "outputId": "ef43873b-0fa0-46a3-ad94-34f497dcca99"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ref                                                                                deadline             category             reward  teamCount  userHasEntered  \n",
            "---------------------------------------------------------------------------------  -------------------  ---------------  ----------  ---------  --------------  \n",
            "https://www.kaggle.com/competitions/ai-mathematical-olympiad-prize                 2024-06-27 23:59:00  Featured         $1,048,576        550           False  \n",
            "https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability        2024-05-27 23:59:00  Featured           $105,000       3364           False  \n",
            "https://www.kaggle.com/competitions/lmsys-chatbot-arena                            2024-08-05 23:59:00  Research           $100,000        228           False  \n",
            "https://www.kaggle.com/competitions/learning-agency-lab-automated-essay-scoring-2  2024-07-02 23:59:00  Featured            $50,000       1317           False  \n",
            "https://www.kaggle.com/competitions/leash-BELKA                                    2024-07-08 23:59:00  Featured            $50,000        836           False  \n",
            "https://www.kaggle.com/competitions/image-matching-challenge-2024                  2024-06-03 23:59:00  Research            $50,000        784           False  \n",
            "https://www.kaggle.com/competitions/birdclef-2024                                  2024-06-10 23:59:00  Research            $50,000        602           False  \n",
            "https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim            2024-07-01 23:59:00  Research            $50,000        301           False  \n",
            "https://www.kaggle.com/competitions/uspto-explainable-ai                           2024-07-24 23:59:00  Featured            $50,000         86           False  \n",
            "https://www.kaggle.com/competitions/playground-series-s4e5                         2024-05-31 23:59:00  Playground             Swag       1226           False  \n",
            "https://www.kaggle.com/competitions/planttraits2024                                2024-06-15 22:00:00  Research          Knowledge        288           False  \n",
            "https://www.kaggle.com/competitions/geolifeclef-2024                               2024-05-24 23:59:00  Research          Knowledge         86           False  \n",
            "https://www.kaggle.com/competitions/titanic                                        2030-01-01 00:00:00  Getting Started   Knowledge      15593            True  \n",
            "https://www.kaggle.com/competitions/house-prices-advanced-regression-techniques    2030-01-01 00:00:00  Getting Started   Knowledge       4622           False  \n",
            "https://www.kaggle.com/competitions/spaceship-titanic                              2030-01-01 00:00:00  Getting Started   Knowledge       2593           False  \n",
            "https://www.kaggle.com/competitions/digit-recognizer                               2030-01-01 00:00:00  Getting Started   Knowledge       1924           False  \n",
            "https://www.kaggle.com/competitions/nlp-getting-started                            2030-01-01 00:00:00  Getting Started   Knowledge       1111           False  \n",
            "https://www.kaggle.com/competitions/store-sales-time-series-forecasting            2030-06-30 23:59:00  Getting Started   Knowledge        719           False  \n",
            "https://www.kaggle.com/competitions/connectx                                       2030-01-01 00:00:00  Getting Started   Knowledge        209           False  \n",
            "https://www.kaggle.com/competitions/gan-getting-started                            2030-07-01 23:59:00  Getting Started   Knowledge        120           False  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "from bs4 import BeautifulSoup\n",
        "import dspy\n",
        "\n",
        "website_url = \"https://www.kaggle.com/competitions\"\n",
        "\n",
        "#website_url = \"https://www.kaggle.com/competitions/playground-series-s4e5/\"\n",
        "\n",
        "res = requests.get(website_url)\n",
        "soup = BeautifulSoup(res.text, 'html.parser')\n",
        "raw_text = [p.text for p in soup.find_all('p') if p.text]"
      ],
      "metadata": {
        "id": "kKUDCUq_gYA1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# now show some of the extracted text\n",
        "raw_text[:10]\n",
        "\n",
        "print(soup)"
      ],
      "metadata": {
        "id": "kSAgU4hxjdZR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "from bs4 import BeautifulSoup\n",
        "import dspy\n",
        "res = requests.get(\"https://grugbrain.dev/\")\n",
        "soup = BeautifulSoup(res.text, 'html.parser')\n",
        "raw_text = [p.text for p in soup.find_all('p') if p.text]"
      ],
      "metadata": {
        "id": "V0rOMfbstuNJ"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "raw_text[:10]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "10-I8gPRw6b4",
        "outputId": "d293e158-0424-433d-cfc8-4dc4c92d4756"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['this collection of thoughts on software development gathered by grug brain developer',\n",
              " 'grug brain developer not so smart, but grug brain developer program many long year and learn some things\\nalthough mostly still confused',\n",
              " 'grug brain developer try collect learns into small, easily digestible and funny page, not only for you, the young grug, but also for him\\nbecause as grug brain developer get older he forget important things, like what had for breakfast or if put pants on',\n",
              " 'big brained developers are many, and some not expected to like this, make sour face',\n",
              " 'THINK they are big brained developers many, many more, and more even definitely probably maybe not like this, many\\nsour face (such is internet)',\n",
              " '(note: grug once think big brained but learn hard way)',\n",
              " 'is fine!',\n",
              " 'is free country sort of and end of day not really matter too much, but grug hope you fun reading and maybe learn from\\nmany, many mistake grug make over long program life',\n",
              " 'apex predator of grug is complexity',\n",
              " 'complexity bad']"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install python-dotenv"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sdvi6NS50oKu",
        "outputId": "ed2c8b3b-5d6c-48c0-ea63-7e43fd915e0b"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting python-dotenv\n",
            "  Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)\n",
            "Installing collected packages: python-dotenv\n",
            "Successfully installed python-dotenv-1.0.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#load .env file\n",
        "from dotenv import load_dotenv\n",
        "load_dotenv()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TZqV28k_1Nr4",
        "outputId": "76a624cb-f0b5-452a-99eb-1cbb75af6d7d"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from openai import OpenAI\n",
        "client = OpenAI()\n",
        "openai_model_name= \"gpt-3.5-turbo\"\n",
        "class BuildMessages:\n",
        "    def __init__(self, system_prompt, user_prompt):\n",
        "        self.system_prompt = system_prompt\n",
        "        self.user_prompt = user_prompt\n",
        "    def render(self, **kwargs):\n",
        "        sys = self.system_prompt.format(**kwargs)\n",
        "        user = self.user_prompt.format(**kwargs)\n",
        "        return [\n",
        "            {\"role\":\"system\", \"content\":sys},\n",
        "            {\"role\":\"user\", \"content\":user},\n",
        "        ]\n",
        "from functools import cache\n",
        "@cache\n",
        "def translate_grug(grug_text):\n",
        "    prompt = BuildMessages(\n",
        "    \"You are an expert in deciphering strange text. The user will provide text written by someone named Grug and you will provide the translation.\",\n",
        "    \"\"\"Translate the following text into plain english: '{text}'.\n",
        "\n",
        "    Do not respond with any other text. Only provide that text. Now take a deep breath and begin.\"\"\"\n",
        ")\n",
        "    result = client.chat.completions.create(messages=prompt.render(text=grug_text), model=openai_model_name)\n",
        "    return result.choices[0].message.content"
      ],
      "metadata": {
        "id": "HFLOcp-xxE30"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# translate dataset\n",
        "dataset = []\n",
        "for grug_text in raw_text[:10]:\n",
        "    translated = translate_grug(grug_text)\n",
        "    dataset.append({\"grug_text\":grug_text, \"plain_english\":translated})"
      ],
      "metadata": {
        "id": "FpP0TwlF1Xbw"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "examples = []\n",
        "for row in dataset:\n",
        "    examples.append(dspy.Example(grug_text=row[\"grug_text\"], plain_english=row[\"plain_english\"]).with_inputs(\"plain_english\"))"
      ],
      "metadata": {
        "id": "jNTkUEjS1mo6"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The key insight here is that we're creating a set of Examples that DSPy is going to use on our behalf to do future translations.\n",
        "\n",
        "We're specifying an input of \"plain english\" and getting back grug text.\n",
        "\n",
        "We tell DSPy about the inputs for our examples by calling with_inputs(\"plain_english\").\n",
        "We can then split the examples into training and test sets:"
      ],
      "metadata": {
        "id": "0UP7Hovj10pX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "from random import shuffle\n",
        "def split_for_train_test(values, test_size = 1/3.0):\n",
        "    shuffle(values)\n",
        "    train = int(len(values)-test_size*len(values))\n",
        "    print(train)\n",
        "    return values[:train], values[train:]\n",
        "train, test = split_for_train_test(examples)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ivhxH0wH1x0U",
        "outputId": "8dba5df5-d767-41c1-84b0-d30dab669e13"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "6\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "train[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "05dqCYuG19lC",
        "outputId": "c5a021da-3cd1-4f25-f5d8-7929be41e2c3"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Example({'grug_text': 'big brained developers are many, and some not expected to like this, make sour face', 'plain_english': 'Skilled developers are abundant, and some may not be happy about this, causing them to frown.'}) (input_keys={'plain_english'})"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Think of Signatures like task specifications. You've got input, you've got output, and a simple prompt (as the docstring) to describe the task.\n",
        "Let's create that for our Grug text."
      ],
      "metadata": {
        "id": "Gf33guPi2US3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import dspy\n",
        "class GrugTranslation(dspy.Signature):\n",
        "    \"Translate plain english to Grug text.\"\n",
        "    plain_english = dspy.InputField()\n",
        "    grug_text = dspy.OutputField()"
      ],
      "metadata": {
        "id": "XtI-dlJw2NZK"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "Q9LlZKyt2jmF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Lets debug this using dspy"
      ],
      "metadata": {
        "id": "tM3OJt_x2kVc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "turbo = dspy.OpenAI(model='gpt-3.5-turbo', max_tokens=1000)\n",
        "dspy.settings.configure(lm=turbo)\n",
        "from dspy.signatures.signature import signature_to_template\n",
        "grug_translation_as_template = signature_to_template(GrugTranslation)\n",
        "print(str(grug_translation_as_template))\n",
        "print(grug_translation_as_template.query(examples[0]))\n",
        "GrugTranslation.signature\n",
        "GrugTranslation.with_instructions"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 174
        },
        "id": "rdUwolQ42ZF1",
        "outputId": "34552c8e-89b8-4870-f35e-b06b5164f36c"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Template(Translate plain english to Grug text., ['Plain English:', 'Grug Text:'])\n",
            "Plain English: Skilled developers are abundant, and some may not be happy about this, causing them to frown.\n",
            "Grug Text: big brained developers are many, and some not expected to like this, make sour face\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<bound method SignatureMeta.with_instructions of GrugTranslation(plain_english -> grug_text\n",
              "    instructions='Translate plain english to Grug text.'\n",
              "    plain_english = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Plain English:', 'desc': '${plain_english}'})\n",
              "    grug_text = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Grug Text:', 'desc': '${grug_text}'})\n",
              ")>"
            ],
            "text/html": [
              "<div style=\"max-width:800px; border: 1px solid var(--colab-border-color);\"><style>\n",
              "      pre.function-repr-contents {\n",
              "        overflow-x: auto;\n",
              "        padding: 8px 12px;\n",
              "        max-height: 500px;\n",
              "      }\n",
              "\n",
              "      pre.function-repr-contents.function-repr-contents-collapsed {\n",
              "        cursor: pointer;\n",
              "        max-height: 100px;\n",
              "      }\n",
              "    </style>\n",
              "    <pre style=\"white-space: initial; background:\n",
              "         var(--colab-secondary-surface-color); padding: 8px 12px;\n",
              "         border-bottom: 1px solid var(--colab-border-color);\"><b>dspy.signatures.signature.SignatureMeta.with_instructions</b><br/>def with_instructions(cls, instructions: str) -&gt; Type[&#x27;Signature&#x27;]</pre><pre class=\"function-repr-contents function-repr-contents-collapsed\" style=\"\"><a class=\"filepath\" style=\"display:none\" href=\"#\">/usr/local/lib/python3.10/dist-packages/dspy/signatures/signature.py</a>&lt;no docstring&gt;</pre>\n",
              "      <script>\n",
              "      if (google.colab.kernel.accessAllowed && google.colab.files && google.colab.files.view) {\n",
              "        for (const element of document.querySelectorAll('.filepath')) {\n",
              "          element.style.display = 'block'\n",
              "          element.onclick = (event) => {\n",
              "            event.preventDefault();\n",
              "            event.stopPropagation();\n",
              "            google.colab.files.view(element.textContent, 93);\n",
              "          };\n",
              "        }\n",
              "      }\n",
              "      for (const element of document.querySelectorAll('.function-repr-contents')) {\n",
              "        element.onclick = (event) => {\n",
              "          event.preventDefault();\n",
              "          event.stopPropagation();\n",
              "          element.classList.toggle('function-repr-contents-collapsed');\n",
              "        };\n",
              "      }\n",
              "      </script>\n",
              "      </div>"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Define module\n",
        "\n",
        "DSPy uses modules to encapsulate the logic for a specific task. This module will take plain English text as input and return the corresponding Grug text when we call the forward method."
      ],
      "metadata": {
        "id": "GxTVtqq52une"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class CoT(dspy.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.prog = dspy.ChainOfThought(GrugTranslation)\n",
        "\n",
        "    def forward(self, plain_english):\n",
        "        return self.prog(plain_english=plain_english)\n",
        "c = CoT()"
      ],
      "metadata": {
        "id": "VHlNICmk2xAK"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "What's powerful here is the ChainOfThought class. Rather than manually specifying to the model that we want it to follow chain of thought, we just call that higher level abstraction and DSPy will take care of it for us.\n",
        "\n",
        "This systematic approach allows us to easily experiment, measure performance, and optimize the translation model.\n",
        "Before optimizing it we can run a zero shot forward pass:"
      ],
      "metadata": {
        "id": "LI3A17RD3ANC"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "O6gGzyO-3JrC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "c.forward(\"You should not construct complex systems.\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jItQ1XcB23L2",
        "outputId": "61d943ad-7b66-4e64-f788-2aa97de4ce38"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Prediction(\n",
              "    rationale=\"avoid confusion and keep things simple. We don't want to overwhelm ourselves with intricate structures that may be difficult to manage.\",\n",
              "    grug_text='You no build big big things.'\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Defining metrics\n",
        "\n",
        "One of the most common metrics for readability is the Automated Readability Index (ARI), which is a formula that produces a score that approximates the grade level needed to understand the text."
      ],
      "metadata": {
        "id": "YBZY7tQP3WWi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# https://apps.dtic.mil/sti/tr/pdf/AD0667273.pdf\n",
        "def automated_readability_index(text):\n",
        "    import re\n",
        "    characters = len(re.sub(r'\\s+', '', text)) # Count characters (ignoring whitespace)\n",
        "    words = len(text.split()) # Count words by splitting the text\n",
        "    # Count sentences by finding period, exclamation, or question mark\n",
        "    sentences = len(re.findall(r'[.!?\\n]', text))\n",
        "    # small change is to add a new line character as grug doesn't seem to use punctuation.\n",
        "    if words == 0 or sentences == 0:  # Prevent division by zero\n",
        "        return 0\n",
        "    # Calculate the Automated Readability Index (ARI)\n",
        "    ari = (4.71 * (characters / words)) + (0.5 * (words / sentences)) - 21.43\n",
        "\n",
        "    return round(ari, 2)"
      ],
      "metadata": {
        "id": "K0rv5O613OsR"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for ex in examples:\n",
        "    source_ari = automated_readability_index(ex.plain_english)\n",
        "    grug_ari = automated_readability_index(ex.grug_text)\n",
        "    print(f\"ARI {source_ari} => {grug_ari}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7HL2OMLZ3orz",
        "outputId": "5cc9c41a-fc42-4306-9a30-dbf75c6a9d32"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ARI 9.53 => 0\n",
            "ARI 8.13 => 14.12\n",
            "ARI 10.24 => 0\n",
            "ARI 14.42 => 0\n",
            "ARI 8.68 => 22.95\n",
            "ARI 9.14 => 14.62\n",
            "ARI 47.58 => -3.95\n",
            "ARI 14.02 => 13.98\n",
            "ARI 7.45 => 0\n",
            "ARI 4.58 => 0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Another important metric we'll consider is semantic similarity, which measures how closely the meaning of the translated text matches the original.\n",
        "Let's do that now and use AI to assess the output as well."
      ],
      "metadata": {
        "id": "BAcCAbI23rSb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# https://dspy-docs.vercel.app/docs/building-blocks/metrics#intermediate-using-ai-feedback-for-your-metric\n",
        "class AssessBasedOnQuestion(dspy.Signature):\n",
        "    \"\"\"Given the assessed text provide a yes or no to the assessment question.\"\"\"\n",
        "    assessed_text = dspy.InputField(format=str)\n",
        "    assessment_question = dspy.InputField(format=str)\n",
        "    assessment_answer = dspy.OutputField(desc=\"Yes or No\")"
      ],
      "metadata": {
        "id": "IebfLZrJ3oy8"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "example_question_assessment = dspy.Example(assessed_text=\"This is a test.\", assessment_question=\"Is this a test?\", assessment_answer=\"Yes\").with_inputs(\"assessed_text\", \"assessment_question\")\n",
        "print(signature_to_template(AssessBasedOnQuestion).query(example_question_assessment))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rQGCuNCw3x5l",
        "outputId": "a3ef3c6a-f672-4862-f02e-834026fae512"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Assessed Text: This is a test.\n",
            "Assessment Question: Is this a test?\n",
            "Assessment Answer: Yes\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note: the example_question_assessment object is technically a Prediction object, but it mirrors the functionality of an Example.\n",
        "Now we can actually define a similarity metric. This metric takes in a truth and a prediction and uses AI feedback to assess the semantic similarity between the two texts.\n",
        "We're using GPT4-Turbo here, but you could use any model you have access to."
      ],
      "metadata": {
        "id": "BOY58Apk31ra"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "gpt4T = dspy.OpenAI(model='gpt-4-turbo', max_tokens=500)\n",
        "def similarity_metric(truth, pred, trace=None):\n",
        "    truth_grug_text = truth.grug_text\n",
        "    proposed_grug_text = pred.grug_text\n",
        "    similarity_question = f\"\"\"Does the assessed text have the same meaning as the gold_standard text provided?\n",
        "Gold Standard: \"{truth_grug_text}\"\n",
        "Provide only a yes or no answer.\"\"\"\n",
        "    with dspy.context(lm=gpt4T):\n",
        "        assessor = dspy.Predict(AssessBasedOnQuestion)\n",
        "        raw_similarity_result = assessor(assessed_text=proposed_grug_text, assessment_question=similarity_question)\n",
        "    print(raw_similarity_result) # for debugging\n",
        "    raw_similarity = raw_similarity_result.assessment_answer.lower().strip()\n",
        "    same_meaning = raw_similarity == 'yes'\n",
        "    return same_meaning"
      ],
      "metadata": {
        "id": "JUIA3Wde35l0"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You'll notice that we had to specify a truth and a pred parameter. This is the standard inteface for any metric that DSPy will use for optimization."
      ],
      "metadata": {
        "id": "sVmgj2Zu4MbA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def ari_metric(truth, pred, trace=None):\n",
        "    truth_grug_text = truth.grug_text\n",
        "    proposed_grug_text = pred.grug_text\n",
        "\n",
        "    gold_ari = automated_readability_index(truth_grug_text)\n",
        "    pred_ari = automated_readability_index(proposed_grug_text)\n",
        "    print(f\"ARI {gold_ari} => {pred_ari}\")\n",
        "    ari_result = pred_ari <= 7.01\n",
        "    return ari_result"
      ],
      "metadata": {
        "id": "3nO4NRwT4NaK"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def overall_metric(provided_example, predicted, trace=None):\n",
        "    similarity = similarity_metric(provided_example, predicted, trace)\n",
        "    ari = ari_metric(provided_example, predicted, trace)\n",
        "    if similarity and ari:\n",
        "        return True\n",
        "    return False"
      ],
      "metadata": {
        "id": "yDLvZ5TX4QJn"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can now use optimization techniques like few-shot learning to fine-tune our models and improve their performance.\n",
        "This allows us to take a more systematic, data-driven approach to working with language models."
      ],
      "metadata": {
        "id": "xfnH3iPF4ZDX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from dspy.teleprompt import BootstrapFewShot\n",
        "config = dict(max_bootstrapped_demos=4, max_labeled_demos=4)\n",
        "optimizer = BootstrapFewShot(metric=overall_metric, **config)\n",
        "optimizer.max_errors = 1 # helpful to debug errors faster\n",
        "optimized_cot = optimizer.compile(CoT(), trainset=train, valset=test)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nUuWYFGS4aiH",
        "outputId": "4981e263-648c-4df5-8567-af28cfa74918"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            " 17%|█▋        | 1/6 [00:01<00:08,  1.67s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prediction(\n",
            "    assessment_answer='Assessment Answer: Yes'\n",
            ")\n",
            "ARI 0 => 0\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r 33%|███▎      | 2/6 [00:03<00:07,  1.80s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prediction(\n",
            "    assessment_answer='Assessment Answer: Yes'\n",
            ")\n",
            "ARI 14.12 => 4.94\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r 50%|█████     | 3/6 [00:06<00:07,  2.35s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prediction(\n",
            "    assessment_answer='Assessed Text: complexity ultimate enemy for Grug\\nAssessment Question: Does the assessed text have the same meaning as the gold_standard text provided?\\nGold Standard: \"apex predator of grug is complexity\"\\nAssessment Answer: No'\n",
            ")\n",
            "ARI 0 => 0\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r 67%|██████▋   | 4/6 [00:09<00:04,  2.42s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prediction(\n",
            "    assessment_answer='Assessed Text: Grug software developer collect thoughts on software development\\nAssessment Question: Does the assessed text have the same meaning as the gold_standard text provided?\\nGold Standard: \"this collection of thoughts on software development gathered by grug brain developer\"\\nAssessment Answer: Yes'\n",
            ")\n",
            "ARI 0 => 0\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r 83%|████████▎ | 5/6 [00:11<00:02,  2.26s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prediction(\n",
            "    assessment_answer='Yes'\n",
            ")\n",
            "ARI 22.95 => 7.19\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 6/6 [00:13<00:00,  2.19s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prediction(\n",
            "    assessment_answer='Yes'\n",
            ")\n",
            "ARI 14.62 => 7.74\n",
            "Bootstrapped 0 full traces after 6 examples in round 0.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Evaluation\n",
        "\n"
      ],
      "metadata": {
        "id": "rR-mjd4T4j-q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from dspy.evaluate import Evaluate\n",
        "individual_metrics = [similarity_metric, ari_metric]\n",
        "for metric in individual_metrics:\n",
        "    evaluate = Evaluate(metric=metric, devset=train, num_threads=1, display_progress=True, display_table=5)\n",
        "    evaluate(optimized_cot)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "q3IlRygu4luh",
        "outputId": "9aa82f80-7326-4cc3-b95e-01b634e6ef33"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Average Metric: 0 / 1  (0.0):  17%|█▋        | 1/6 [00:02<00:10,  2.03s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prediction(\n",
            "    assessment_answer='Assessment Answer: Yes'\n",
            ")\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Average Metric: 0 / 2  (0.0):  33%|███▎      | 2/6 [00:04<00:08,  2.03s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prediction(\n",
            "    assessment_answer='Assessment Answer: No'\n",
            ")\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Average Metric: 0 / 3  (0.0):  50%|█████     | 3/6 [00:08<00:08,  2.97s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prediction(\n",
            "    assessment_answer='Assessed Text: grug think complexity ultimate predator\\nAssessment Question: Does the assessed text have the same meaning as the gold_standard text provided?\\nGold Standard: \"apex predator of grug is complexity\"\\nAssessment Answer: Yes'\n",
            ")\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Average Metric: 0 / 4  (0.0):  67%|██████▋   | 4/6 [00:09<00:04,  2.38s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prediction(\n",
            "    assessment_answer='Assessment Answer: Yes'\n",
            ")\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Average Metric: 0 / 5  (0.0):  83%|████████▎ | 5/6 [00:15<00:03,  3.71s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prediction(\n",
            "    assessment_answer='Assessed Text: grug brain developer try collect lesson small, easy funny page not just for young grug but also for self. as grug brain developer age forget important thing like breakfast or pants.\\nAssessment Question: Does the assessed text have the same meaning as the gold_standard text provided?\\nGold Standard: \"grug brain developer try collect learns into small, easily digestible and funny page, not only for you, the young grug, but also for him because as grug brain developer get older he forget important things, like what had for breakfast or if put pants on\"\\nAssessment Answer: Yes'\n",
            ")\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Average Metric: 1 / 6  (16.7): 100%|██████████| 6/6 [00:17<00:00,  2.93s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prediction(\n",
            "    assessment_answer='Yes'\n",
            ")\n",
            "Average Metric: 1 / 6  (16.7%)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7ecfb39dcbb0>"
            ],
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_8b1d6 th {\n",
              "  text-align: left;\n",
              "}\n",
              "#T_8b1d6 td {\n",
              "  text-align: left;\n",
              "}\n",
              "#T_8b1d6_row0_col0, #T_8b1d6_row0_col1, #T_8b1d6_row0_col2, #T_8b1d6_row0_col3, #T_8b1d6_row0_col4, #T_8b1d6_row1_col0, #T_8b1d6_row1_col1, #T_8b1d6_row1_col2, #T_8b1d6_row1_col3, #T_8b1d6_row1_col4, #T_8b1d6_row2_col0, #T_8b1d6_row2_col1, #T_8b1d6_row2_col2, #T_8b1d6_row2_col3, #T_8b1d6_row2_col4, #T_8b1d6_row3_col0, #T_8b1d6_row3_col1, #T_8b1d6_row3_col2, #T_8b1d6_row3_col3, #T_8b1d6_row3_col4, #T_8b1d6_row4_col0, #T_8b1d6_row4_col1, #T_8b1d6_row4_col2, #T_8b1d6_row4_col3, #T_8b1d6_row4_col4 {\n",
              "  text-align: left;\n",
              "  white-space: pre-wrap;\n",
              "  word-wrap: break-word;\n",
              "  max-width: 400px;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_8b1d6\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_8b1d6_level0_col0\" class=\"col_heading level0 col0\" >example_grug_text</th>\n",
              "      <th id=\"T_8b1d6_level0_col1\" class=\"col_heading level0 col1\" >plain_english</th>\n",
              "      <th id=\"T_8b1d6_level0_col2\" class=\"col_heading level0 col2\" >rationale</th>\n",
              "      <th id=\"T_8b1d6_level0_col3\" class=\"col_heading level0 col3\" >pred_grug_text</th>\n",
              "      <th id=\"T_8b1d6_level0_col4\" class=\"col_heading level0 col4\" >similarity_metric</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_8b1d6_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_8b1d6_row0_col0\" class=\"data row0 col0\" >big brained developers are many, and some not expected to like this, make sour face</td>\n",
              "      <td id=\"T_8b1d6_row0_col1\" class=\"data row0 col1\" >Skilled developers are abundant, and some may not be happy about this, causing them to frown.</td>\n",
              "      <td id=\"T_8b1d6_row0_col2\" class=\"data row0 col2\" >produce the grug_text. We start with the idea that skilled developers are many, then we consider that some are not happy, leading to the action...</td>\n",
              "      <td id=\"T_8b1d6_row0_col3\" class=\"data row0 col3\" >many skilled developer, some not happy, make frown</td>\n",
              "      <td id=\"T_8b1d6_row0_col4\" class=\"data row0 col4\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_8b1d6_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "      <td id=\"T_8b1d6_row1_col0\" class=\"data row1 col0\" >THINK they are big brained developers many, many more, and more even definitely probably maybe not like this, many\n",
              "sour face (such is internet)</td>\n",
              "      <td id=\"T_8b1d6_row1_col1\" class=\"data row1 col1\" >I believe there are many developers who think they are highly intelligent, but in reality, there are probably many others who are not like this...</td>\n",
              "      <td id=\"T_8b1d6_row1_col2\" class=\"data row1 col2\" >produce the grug_text. We will break down the idea of developers thinking they are smart, then contrast it with the reality of others who are...</td>\n",
              "      <td id=\"T_8b1d6_row1_col3\" class=\"data row1 col3\" >many developer think big brain but many not, internet bad sour</td>\n",
              "      <td id=\"T_8b1d6_row1_col4\" class=\"data row1 col4\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_8b1d6_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "      <td id=\"T_8b1d6_row2_col0\" class=\"data row2 col0\" >apex predator of grug is complexity</td>\n",
              "      <td id=\"T_8b1d6_row2_col1\" class=\"data row2 col1\" >Grug considers complexity to be the ultimate predator.</td>\n",
              "      <td id=\"T_8b1d6_row2_col2\" class=\"data row2 col2\" >produce the Grug text. We start with the subject, Grug, then move on to his belief about complexity being the ultimate predator.</td>\n",
              "      <td id=\"T_8b1d6_row2_col3\" class=\"data row2 col3\" >grug think complexity ultimate predator</td>\n",
              "      <td id=\"T_8b1d6_row2_col4\" class=\"data row2 col4\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_8b1d6_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
              "      <td id=\"T_8b1d6_row3_col0\" class=\"data row3 col0\" >this collection of thoughts on software development gathered by grug brain developer</td>\n",
              "      <td id=\"T_8b1d6_row3_col1\" class=\"data row3 col1\" >Grug the software developer has gathered this collection of thoughts on software development.</td>\n",
              "      <td id=\"T_8b1d6_row3_col2\" class=\"data row3 col2\" >produce the grug_text. We will break down the sentence and simplify it for Grug to understand.</td>\n",
              "      <td id=\"T_8b1d6_row3_col3\" class=\"data row3 col3\" >grug software developer collect thoughts on software development</td>\n",
              "      <td id=\"T_8b1d6_row3_col4\" class=\"data row3 col4\" >False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_8b1d6_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
              "      <td id=\"T_8b1d6_row4_col0\" class=\"data row4 col0\" >grug brain developer try collect learns into small, easily digestible and funny page, not only for you, the young grug, but also for him because...</td>\n",
              "      <td id=\"T_8b1d6_row4_col1\" class=\"data row4 col1\" >Grug, a brain developer, is trying to collect lessons into small, easily digestible, and funny pages. This is not only for the young Grug, but...</td>\n",
              "      <td id=\"T_8b1d6_row4_col2\" class=\"data row4 col2\" >produce the grug_text. We will break down the key points and simplify them for Grug's understanding.</td>\n",
              "      <td id=\"T_8b1d6_row4_col3\" class=\"data row4 col3\" >grug brain developer try collect lesson small, easy funny page not just for young grug but also for self. as grug brain developer age forget...</td>\n",
              "      <td id=\"T_8b1d6_row4_col4\" class=\"data row4 col4\" >False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "                <div style='\n",
              "                    text-align: center;\n",
              "                    font-size: 16px;\n",
              "                    font-weight: bold;\n",
              "                    color: #555;\n",
              "                    margin: 10px 0;'>\n",
              "                    ... 1 more rows not displayed ...\n",
              "                </div>\n",
              "                "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Average Metric: 5 / 6  (83.3): 100%|██████████| 6/6 [00:00<00:00, 318.20it/s]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ARI 0 => 0\n",
            "ARI 14.12 => 0\n",
            "ARI 0 => 0\n",
            "ARI 0 => 0\n",
            "ARI 22.95 => 9.26\n",
            "ARI 14.62 => 0\n",
            "Average Metric: 5 / 6  (83.3%)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7ecfa3571480>"
            ],
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_8113a th {\n",
              "  text-align: left;\n",
              "}\n",
              "#T_8113a td {\n",
              "  text-align: left;\n",
              "}\n",
              "#T_8113a_row0_col0, #T_8113a_row0_col1, #T_8113a_row0_col2, #T_8113a_row0_col3, #T_8113a_row0_col4, #T_8113a_row1_col0, #T_8113a_row1_col1, #T_8113a_row1_col2, #T_8113a_row1_col3, #T_8113a_row1_col4, #T_8113a_row2_col0, #T_8113a_row2_col1, #T_8113a_row2_col2, #T_8113a_row2_col3, #T_8113a_row2_col4, #T_8113a_row3_col0, #T_8113a_row3_col1, #T_8113a_row3_col2, #T_8113a_row3_col3, #T_8113a_row3_col4, #T_8113a_row4_col0, #T_8113a_row4_col1, #T_8113a_row4_col2, #T_8113a_row4_col3, #T_8113a_row4_col4 {\n",
              "  text-align: left;\n",
              "  white-space: pre-wrap;\n",
              "  word-wrap: break-word;\n",
              "  max-width: 400px;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_8113a\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_8113a_level0_col0\" class=\"col_heading level0 col0\" >example_grug_text</th>\n",
              "      <th id=\"T_8113a_level0_col1\" class=\"col_heading level0 col1\" >plain_english</th>\n",
              "      <th id=\"T_8113a_level0_col2\" class=\"col_heading level0 col2\" >rationale</th>\n",
              "      <th id=\"T_8113a_level0_col3\" class=\"col_heading level0 col3\" >pred_grug_text</th>\n",
              "      <th id=\"T_8113a_level0_col4\" class=\"col_heading level0 col4\" >ari_metric</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_8113a_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_8113a_row0_col0\" class=\"data row0 col0\" >big brained developers are many, and some not expected to like this, make sour face</td>\n",
              "      <td id=\"T_8113a_row0_col1\" class=\"data row0 col1\" >Skilled developers are abundant, and some may not be happy about this, causing them to frown.</td>\n",
              "      <td id=\"T_8113a_row0_col2\" class=\"data row0 col2\" >produce the grug_text. We start with the idea that skilled developers are many, then we consider that some are not happy, leading to the action...</td>\n",
              "      <td id=\"T_8113a_row0_col3\" class=\"data row0 col3\" >many skilled developer, some not happy, make frown</td>\n",
              "      <td id=\"T_8113a_row0_col4\" class=\"data row0 col4\" >✔️ [True]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_8113a_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "      <td id=\"T_8113a_row1_col0\" class=\"data row1 col0\" >THINK they are big brained developers many, many more, and more even definitely probably maybe not like this, many\n",
              "sour face (such is internet)</td>\n",
              "      <td id=\"T_8113a_row1_col1\" class=\"data row1 col1\" >I believe there are many developers who think they are highly intelligent, but in reality, there are probably many others who are not like this...</td>\n",
              "      <td id=\"T_8113a_row1_col2\" class=\"data row1 col2\" >produce the grug_text. We will break down the idea of developers thinking they are smart, then contrast it with the reality of others who are...</td>\n",
              "      <td id=\"T_8113a_row1_col3\" class=\"data row1 col3\" >many developer think big brain but many not, internet bad sour</td>\n",
              "      <td id=\"T_8113a_row1_col4\" class=\"data row1 col4\" >✔️ [True]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_8113a_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "      <td id=\"T_8113a_row2_col0\" class=\"data row2 col0\" >apex predator of grug is complexity</td>\n",
              "      <td id=\"T_8113a_row2_col1\" class=\"data row2 col1\" >Grug considers complexity to be the ultimate predator.</td>\n",
              "      <td id=\"T_8113a_row2_col2\" class=\"data row2 col2\" >produce the Grug text. We start with the subject, Grug, then move on to his belief about complexity being the ultimate predator.</td>\n",
              "      <td id=\"T_8113a_row2_col3\" class=\"data row2 col3\" >grug think complexity ultimate predator</td>\n",
              "      <td id=\"T_8113a_row2_col4\" class=\"data row2 col4\" >✔️ [True]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_8113a_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
              "      <td id=\"T_8113a_row3_col0\" class=\"data row3 col0\" >this collection of thoughts on software development gathered by grug brain developer</td>\n",
              "      <td id=\"T_8113a_row3_col1\" class=\"data row3 col1\" >Grug the software developer has gathered this collection of thoughts on software development.</td>\n",
              "      <td id=\"T_8113a_row3_col2\" class=\"data row3 col2\" >produce the grug_text. We will break down the sentence and simplify it for Grug to understand.</td>\n",
              "      <td id=\"T_8113a_row3_col3\" class=\"data row3 col3\" >grug software developer collect thoughts on software development</td>\n",
              "      <td id=\"T_8113a_row3_col4\" class=\"data row3 col4\" >✔️ [True]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_8113a_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
              "      <td id=\"T_8113a_row4_col0\" class=\"data row4 col0\" >grug brain developer try collect learns into small, easily digestible and funny page, not only for you, the young grug, but also for him because...</td>\n",
              "      <td id=\"T_8113a_row4_col1\" class=\"data row4 col1\" >Grug, a brain developer, is trying to collect lessons into small, easily digestible, and funny pages. This is not only for the young Grug, but...</td>\n",
              "      <td id=\"T_8113a_row4_col2\" class=\"data row4 col2\" >produce the grug_text. We will break down the key points and simplify them for Grug's understanding.</td>\n",
              "      <td id=\"T_8113a_row4_col3\" class=\"data row4 col3\" >grug brain developer try collect lesson small, easy funny page not just for young grug but also for self. as grug brain developer age forget...</td>\n",
              "      <td id=\"T_8113a_row4_col4\" class=\"data row4 col4\" >False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "                <div style='\n",
              "                    text-align: center;\n",
              "                    font-size: 16px;\n",
              "                    font-weight: bold;\n",
              "                    color: #555;\n",
              "                    margin: 10px 0;'>\n",
              "                    ... 1 more rows not displayed ...\n",
              "                </div>\n",
              "                "
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Manual inspection"
      ],
      "metadata": {
        "id": "mQixBsci4rOI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "optimized_cot.forward(\"You should not construct complex systems.\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tCp2pu-_4tU9",
        "outputId": "c01bc18e-a644-442c-ac21-eeb922ad6f13"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Prediction(\n",
              "    rationale='avoid confusion and keep things simple.',\n",
              "    grug_text='no build complex system'\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# save model\n",
        "optimized_cot.save(path=\"/tmp/model.json\")"
      ],
      "metadata": {
        "id": "FWEUvLI440GD"
      },
      "execution_count": 25,
      "outputs": []
    }
  ]
}