06_finetuning/06_part-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "12e91914-5f51-43fa-b65b-625e73b4d17b",
   "metadata": {
    "id": "12e91914-5f51-43fa-b65b-625e73b4d17b"
   },
   "source": [
    "**LLM Workshop 2024 by Sebastian Raschka**\n",
    "\n",
    "This code is based on *Build a Large Language Model (From Scratch)*, [https://github.com/rasbt/LLMs-from-scratch](https://github.com/rasbt/LLMs-from-scratch)\n",
    "\n",
    "- Instruction finetuning from scratch: [ch07.ipynb](https://github.com/rasbt/LLMs-from-scratch/blob/main/ch07/01_main-chapter-code/ch07.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2520ec3-722f-4f44-bdd1-885b13e7afbf",
   "metadata": {
    "id": "c2520ec3-722f-4f44-bdd1-885b13e7afbf"
   },
   "source": [
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "# 6) Instruction finetuning (part 1; intro)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "264fca98-2f9a-4193-b435-2abfa3b4142f",
   "metadata": {
    "id": "264fca98-2f9a-4193-b435-2abfa3b4142f"
   },
   "source": [
    "<img src=\"figures/01.png\" width=1000px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bbc68e9-75b3-41f1-ac2c-e071c3cd0813",
   "metadata": {
    "id": "8bbc68e9-75b3-41f1-ac2c-e071c3cd0813"
   },
   "source": [
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "# 6.1 Introduction to instruction finetuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53dba24a-6805-496c-9a7f-c75e2d3527ab",
   "metadata": {
    "id": "53dba24a-6805-496c-9a7f-c75e2d3527ab"
   },
   "source": [
    "- We saw that pretraining an LLM involves a training procedure where it learns to generate one word at a time\n",
    "- Hence, a pretrained LLM is good at text completion, but it is not good at following instructions\n",
    "- In this last part of the workshop, we teach the LLM to follow instructions better"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18dc0535-0904-44ed-beaf-9b678292ef35",
   "metadata": {
    "id": "18dc0535-0904-44ed-beaf-9b678292ef35"
   },
   "source": [
    "<img src=\"figures/02.png\" width=800px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5384f0cf-ef3c-4436-a5fa-59bd25649f86",
   "metadata": {
    "id": "5384f0cf-ef3c-4436-a5fa-59bd25649f86"
   },
   "source": [
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "# 6.2 Preparing a dataset for supervised instruction finetuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "888e4f5a-cf8a-4b59-8c2c-666c3e99f174",
   "metadata": {
    "id": "f8b34ff8-619f-4e89-bd03-ce513269760d"
   },
   "source": [
    "- We will work with a simple instruction dataset I prepared for this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0G3axLw6kY1N",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0G3axLw6kY1N",
    "outputId": "e742b203-78e8-44f4-edbc-8dcaeba3fc5e"
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "\n",
    "file_path = \"LLM-workshop-2024/06_finetuning/instruction-data.json\"\n",
    "\n",
    "with open(file_path, \"r\") as file:\n",
    "    data = json.load(file)\n",
    "print(\"Number of entries:\", len(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7af8176-4255-4e92-8c7d-998771733eb8",
   "metadata": {
    "id": "d7af8176-4255-4e92-8c7d-998771733eb8"
   },
   "source": [
    "- Each item in the `data` list we loaded from the JSON file above is a dictionary in the following form"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "-LiuBMsHkzQV",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-LiuBMsHkzQV",
    "outputId": "535ddaef-ddc8-4a40-96d2-12d7414838b1"
   },
   "outputs": [],
   "source": [
    "print(\"Example entry:\\n\", data[50])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5a32b34-485a-4816-a77a-da14f9fe6e46",
   "metadata": {
    "id": "c5a32b34-485a-4816-a77a-da14f9fe6e46"
   },
   "source": [
    "- Note that the `'input'` field can be empty:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "uFInFxDDk2Je",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uFInFxDDk2Je",
    "outputId": "01c760fd-53d6-4d94-997d-f72c6675c64e"
   },
   "outputs": [],
   "source": [
    "print(\"Another example entry:\\n\", data[999])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f034799a-6575-45fd-98c9-9d1012d0fd58",
   "metadata": {
    "id": "f034799a-6575-45fd-98c9-9d1012d0fd58"
   },
   "source": [
    "- Instruction finetuning is often referred to as \"supervised instruction finetuning\" because it involves training a model on a dataset where the input-output pairs are explicitly provided\n",
    "- There are different ways to format the entries as inputs to the LLM; the figure below illustrates two example formats that were used for training the Alpaca (https://crfm.stanford.edu/2023/03/13/alpaca.html) and Phi-3 (https://arxiv.org/abs/2404.14219) LLMs, respectively"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dffa4f70-44d4-4be4-89a9-2159f4885b10",
   "metadata": {
    "id": "dffa4f70-44d4-4be4-89a9-2159f4885b10"
   },
   "source": [
    "<img src=\"figures/03.png\" width=900px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd79a74e-befb-491c-be49-f777a6a5b6a6",
   "metadata": {
    "id": "dd79a74e-befb-491c-be49-f777a6a5b6a6"
   },
   "source": [
    "- Suppose we use Alpaca-style prompt formatting, which was the original prompt template for instruction finetuning\n",
    "- Shown below is how we format the input that we would pass as input to the LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "Jhk37nnJnkBh",
   "metadata": {
    "id": "Jhk37nnJnkBh"
   },
   "outputs": [],
   "source": [
    "def format_input(entry):\n",
    "    instruction_text = (\n",
    "        f\"Below is an instruction that describes a task. \"\n",
    "        f\"Write a response that appropriately completes the request.\"\n",
    "        f\"\\n\\n### Instruction:\\n{entry['instruction']}\"\n",
    "    )\n",
    "\n",
    "    input_text = f\"\\n\\n### Input:\\n{entry['input']}\" if entry[\"input\"] else \"\"\n",
    "\n",
    "    return instruction_text + input_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "011e78b4-e89a-4653-a2ee-7b2739ca04d6",
   "metadata": {
    "id": "011e78b4-e89a-4653-a2ee-7b2739ca04d6"
   },
   "source": [
    "- A formatted response with input field looks like as shown below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "F9UQRfjzo4Js",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "F9UQRfjzo4Js",
    "outputId": "47ae5684-c906-41bd-a3ea-e1504f861fd1"
   },
   "outputs": [],
   "source": [
    "model_input = format_input(data[50])\n",
    "desired_response = f\"\\n\\n### Response:\\n{data[50]['output']}\"\n",
    "\n",
    "print(model_input + desired_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dc93ddf-431c-49c0-96f2-fb3a79c4d94c",
   "metadata": {
    "id": "4dc93ddf-431c-49c0-96f2-fb3a79c4d94c"
   },
   "source": [
    "- Below is a formatted response without an input field"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3891fa9-f738-41cd-946c-80ef9a99c346",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "a3891fa9-f738-41cd-946c-80ef9a99c346",
    "outputId": "4630975d-9f2e-448a-a899-2f54bdf4a114"
   },
   "outputs": [],
   "source": [
    "model_input = format_input(data[999])\n",
    "desired_response = f\"\\n\\n### Response:\\n{data[999]['output']}\"\n",
    "\n",
    "print(model_input + desired_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9af423f-aad9-4b3c-bea5-153021c04862",
   "metadata": {
    "id": "b9af423f-aad9-4b3c-bea5-153021c04862"
   },
   "source": [
    "- Tokenized, this looks like as follows\n",
    "\n",
    "<img src=\"figures/04.png\" width=1000px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e5bd7bc-f347-4cf8-a0c2-94cb8799e427",
   "metadata": {
    "id": "9e5bd7bc-f347-4cf8-a0c2-94cb8799e427"
   },
   "source": [
    "- To make it work with batches, we add \"padding\" tokens "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65c4d943-4aa8-4a44-874e-05bc6831fbd3",
   "metadata": {
    "id": "65c4d943-4aa8-4a44-874e-05bc6831fbd3"
   },
   "source": [
    "- Tokenized, this looks like as follows\n",
    "\n",
    "<img src=\"figures/05.png\" width=1000px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ff6c2a5-449c-49e7-ba35-402e1cdbb2ef",
   "metadata": {},
   "source": [
    "- Above, only the inputs are shown for simplicity; however, similar to pretraining, the target tokens are shifted by 1 position:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0386b6fe-3455-4e70-becd-a5a4681ba2ef",
   "metadata": {
    "id": "0386b6fe-3455-4e70-becd-a5a4681ba2ef"
   },
   "source": [
    "- Tokenized, this looks like as follows\n",
    "\n",
    "<img src=\"figures/06.png\" width=700px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cef09d21-b652-4760-abea-4f76920e6a25",
   "metadata": {
    "id": "cef09d21-b652-4760-abea-4f76920e6a25"
   },
   "source": [
    "- In addition, it is also common to mask the target text\n",
    "- By default, PyTorch has the `cross_entropy(..., ignore_index=-100)` setting to ignore examples corresponding to the label -100\n",
    "- Using this -100 `ignore_index`, we can ignore the additional end-of-text (padding) tokens in the batches that we used to pad the training examples to equal length\n",
    "- However, we don't want to ignore the first instance of the end-of-text (padding) token (50256) because it can help signal to the LLM when the response is complete"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fab8f0ed-80e8-4fd9-bf84-e5d0e0bc0a39",
   "metadata": {
    "id": "fab8f0ed-80e8-4fd9-bf84-e5d0e0bc0a39"
   },
   "source": [
    "- Tokenized, this looks like as follows\n",
    "\n",
    "<img src=\"figures/07.png\" width=1000px>"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "A100",
   "machine_shape": "hm",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}