Gemma/Finetune_with_LLaMA_Factory.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### 版權所有 2024 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "cellView": "form",
        "id": "tuOe1ymfHZPu"
      },
      "outputs": [],
      "source": [
        "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dfsDR_omdNea"
      },
      "source": [
        "# Gemma - 使用 LLaMA Factory 微調\n",
        "\n",
        "此筆記本展示了如何使用 LLaMA Factory 微調 Gemma。[LLaMA Factory](https://github.com/InternLM/xtuner) 是一個專門為微調 LLMs 設計的工具。LLaMA Factory 封裝了 Hugging Face 的微調功能，並提供了一個簡單的介面來進行微調。使用 LLaMA Factory 微調 Gemma 非常容易。此筆記本非常接近 LLaMA Factory 官方的 [Colab 筆記本](https://colab.research.google.com/drive/1eRTPn37ltBbYsISy9Aw2NuI2Aq5CQrD9?usp=sharing)。\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/doggy8088/gemma-cookbook/blob/zh-tw-240628/Gemma/Finetune_with_LLaMA_Factory.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />在 Google Colab 中執行</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MwMiP7jDdAL1"
      },
      "source": [
        "## 設定\n",
        "\n",
        "### 選擇 Colab 執行環境\n",
        "要完成此指南，你需要有一個資源充足的 Colab 執行環境來執行 Gemma 模型。在這種情況下，你可以使用 T4 GPU：\n",
        "\n",
        "1. 在 Colab 視窗的右上角，選擇 **▾ (其他連接選項)** 。\n",
        "2. 選擇 **更改執行環境類型** 。\n",
        "3. 在 **硬體加速器** 下，選擇 **T4 GPU** 。\n",
        "\n",
        "### 在 Hugging Face 上設定 Gemma\n",
        "LLaMA Factory 在底層使用 Hugging Face。因此你需要：\n",
        "\n",
        "* 通過接受 Hugging Face 上特定模型頁面的 Gemma 許可來獲取 [huggingface.co](huggingface.co) 上的 Gemma 訪問權限，即 [Gemma 2B](https://huggingface.co/google/gemma-2b)。\n",
        "* 生成一個 [Hugging Face 訪問令牌](https://huggingface.co/docs/hub/en/security-tokens) 並將其配置為 Colab 機密 'HF_TOKEN'。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "AVvJYwne3hha"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from google.colab import userdata\n",
        "# Note: `userdata.get` is a Colab API. If you're not using Colab, set the env\n",
        "# vars as appropriate for your system.\n",
        "os.environ[\"HF_TOKEN\"] = userdata.get(\"HF_TOKEN\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8yUF4Hk5dOoz"
      },
      "source": [
        "### 安裝 LLaMA Factory\n",
        "\n",
        "從 GitHub 上的原始碼安裝 LLaMA Factory。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4pY14h6_bDrr",
        "outputId": "14047e59-163c-43ee-e5cd-43759ef34dcf"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "/content\n",
            "Cloning into 'LLaMA-Factory'...\n",
            "remote: Enumerating objects: 12511, done.\u001b[K\n",
            "remote: Counting objects: 100% (1292/1292), done.\u001b[K\n",
            "remote: Compressing objects: 100% (552/552), done.\u001b[K\n",
            "remote: Total 12511 (delta 883), reused 1026 (delta 723), pack-reused 11219\u001b[K\n",
            "Receiving objects: 100% (12511/12511), 218.87 MiB | 13.35 MiB/s, done.\n",
            "Resolving deltas: 100% (9132/9132), done.\n",
            "/content/LLaMA-Factory\n",
            "\u001b[0m\u001b[01;34massets\u001b[0m/       docker-compose.yml  \u001b[01;34mexamples\u001b[0m/  pyproject.toml  requirements.txt  \u001b[01;34msrc\u001b[0m/\n",
            "CITATION.cff  Dockerfile          LICENSE    README.md       \u001b[01;34mscripts\u001b[0m/          \u001b[01;34mtests\u001b[0m/\n",
            "\u001b[01;34mdata\u001b[0m/         \u001b[01;34mevaluation\u001b[0m/         Makefile   README_zh.md    setup.py\n",
            "Obtaining file:///content/LLaMA-Factory\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Checking if build backend supports build_editable ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build editable ... \u001b[?25l\u001b[?25hdone\n",
            "  Installing backend dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing editable metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: transformers>=4.37.2 in /usr/local/lib/python3.10/dist-packages (from llamafactory==0.7.2.dev0) (4.41.1)\n",
            "Collecting datasets>=2.14.3 (from llamafactory==0.7.2.dev0)\n",
            "  Downloading datasets-2.19.1-py3-none-any.whl (542 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m542.0/542.0 kB\u001b[0m \u001b[31m8.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting accelerate>=0.27.2 (from llamafactory==0.7.2.dev0)\n",
            "  Downloading accelerate-0.30.1-py3-none-any.whl (302 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m302.6/302.6 kB\u001b[0m \u001b[31m28.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting peft>=0.10.0 (from llamafactory==0.7.2.dev0)\n",
            "  Downloading peft-0.11.1-py3-none-any.whl (251 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m251.6/251.6 kB\u001b[0m \u001b[31m28.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting trl>=0.8.1 (from llamafactory==0.7.2.dev0)\n",
            "  Downloading trl-0.8.6-py3-none-any.whl (245 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m245.2/245.2 kB\u001b[0m \u001b[31m29.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting gradio>=4.0.0 (from llamafactory==0.7.2.dev0)\n",
            "  Downloading gradio-4.32.2-py3-none-any.whl (12.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.3/12.3 MB\u001b[0m \u001b[31m56.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from llamafactory==0.7.2.dev0) (1.11.4)\n",
            "Collecting einops (from llamafactory==0.7.2.dev0)\n",
            "  Downloading einops-0.8.0-py3-none-any.whl (43 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.2/43.2 kB\u001b[0m \u001b[31m6.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (from llamafactory==0.7.2.dev0) (0.1.99)\n",
            "Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (from llamafactory==0.7.2.dev0) (3.20.3)\n",
            "Collecting uvicorn (from llamafactory==0.7.2.dev0)\n",
            "  Downloading uvicorn-0.30.0-py3-none-any.whl (62 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.4/62.4 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pydantic in /usr/local/lib/python3.10/dist-packages (from llamafactory==0.7.2.dev0) (2.7.2)\n",
            "Collecting fastapi (from llamafactory==0.7.2.dev0)\n",
            "  Downloading fastapi-0.111.0-py3-none-any.whl (91 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m92.0/92.0 kB\u001b[0m \u001b[31m13.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting sse-starlette (from llamafactory==0.7.2.dev0)\n",
            "  Downloading sse_starlette-2.1.0-py3-none-any.whl (9.2 kB)\n",
            "Requirement already satisfied: matplotlib>=3.7.0 in /usr/local/lib/python3.10/dist-packages (from llamafactory==0.7.2.dev0) (3.7.1)\n",
            "Collecting fire (from llamafactory==0.7.2.dev0)\n",
            "  Downloading fire-0.6.0.tar.gz (88 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m88.4/88.4 kB\u001b[0m \u001b[31m13.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from llamafactory==0.7.2.dev0) (24.0)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from llamafactory==0.7.2.dev0) (6.0.1)\n",
            "Collecting bitsandbytes>=0.39.0 (from llamafactory==0.7.2.dev0)\n",
            "  Downloading bitsandbytes-0.43.1-py3-none-manylinux_2_24_x86_64.whl (119.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m119.8/119.8 MB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: torch>=1.13.1 in /usr/local/lib/python3.10/dist-packages (from llamafactory==0.7.2.dev0) (2.3.0+cu121)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.27.2->llamafactory==0.7.2.dev0) (1.25.2)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.27.2->llamafactory==0.7.2.dev0) (5.9.5)\n",
            "Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.27.2->llamafactory==0.7.2.dev0) (0.23.2)\n",
            "Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.27.2->llamafactory==0.7.2.dev0) (0.4.3)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from datasets>=2.14.3->llamafactory==0.7.2.dev0) (3.14.0)\n",
            "Requirement already satisfied: pyarrow>=12.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.14.3->llamafactory==0.7.2.dev0) (14.0.2)\n",
            "Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets>=2.14.3->llamafactory==0.7.2.dev0) (0.6)\n",
            "Collecting dill<0.3.9,>=0.3.0 (from datasets>=2.14.3->llamafactory==0.7.2.dev0)\n",
            "  Downloading dill-0.3.8-py3-none-any.whl (116 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m15.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets>=2.14.3->llamafactory==0.7.2.dev0) (2.0.3)\n",
            "Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.14.3->llamafactory==0.7.2.dev0) (2.31.0)\n",
            "Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.14.3->llamafactory==0.7.2.dev0) (4.66.4)\n",
            "Collecting xxhash (from datasets>=2.14.3->llamafactory==0.7.2.dev0)\n",
            "  Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m16.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting multiprocess (from datasets>=2.14.3->llamafactory==0.7.2.dev0)\n",
            "  Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m16.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: fsspec[http]<=2024.3.1,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.14.3->llamafactory==0.7.2.dev0) (2023.6.0)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets>=2.14.3->llamafactory==0.7.2.dev0) (3.9.5)\n",
            "Collecting aiofiles<24.0,>=22.0 (from gradio>=4.0.0->llamafactory==0.7.2.dev0)\n",
            "  Downloading aiofiles-23.2.1-py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: altair<6.0,>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=4.0.0->llamafactory==0.7.2.dev0) (4.2.2)\n",
            "Collecting ffmpy (from gradio>=4.0.0->llamafactory==0.7.2.dev0)\n",
            "  Downloading ffmpy-0.3.2.tar.gz (5.5 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting gradio-client==0.17.0 (from gradio>=4.0.0->llamafactory==0.7.2.dev0)\n",
            "  Downloading gradio_client-0.17.0-py3-none-any.whl (316 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m316.3/316.3 kB\u001b[0m \u001b[31m33.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting httpx>=0.24.1 (from gradio>=4.0.0->llamafactory==0.7.2.dev0)\n",
            "  Downloading httpx-0.27.0-py3-none-any.whl (75 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m12.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: importlib-resources<7.0,>=1.3 in /usr/local/lib/python3.10/dist-packages (from gradio>=4.0.0->llamafactory==0.7.2.dev0) (6.4.0)\n",
            "Requirement already satisfied: jinja2<4.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=4.0.0->llamafactory==0.7.2.dev0) (3.1.4)\n",
            "Requirement already satisfied: markupsafe~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=4.0.0->llamafactory==0.7.2.dev0) (2.1.5)\n",
            "Collecting orjson~=3.0 (from gradio>=4.0.0->llamafactory==0.7.2.dev0)\n",
            "  Downloading orjson-3.10.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m142.5/142.5 kB\u001b[0m \u001b[31m19.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pillow<11.0,>=8.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=4.0.0->llamafactory==0.7.2.dev0) (9.4.0)\n",
            "Collecting pydub (from gradio>=4.0.0->llamafactory==0.7.2.dev0)\n",
            "  Downloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)\n",
            "Collecting python-multipart>=0.0.9 (from gradio>=4.0.0->llamafactory==0.7.2.dev0)\n",
            "  Downloading python_multipart-0.0.9-py3-none-any.whl (22 kB)\n",
            "Collecting ruff>=0.2.2 (from gradio>=4.0.0->llamafactory==0.7.2.dev0)\n",
            "  Downloading ruff-0.4.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.8/8.8 MB\u001b[0m \u001b[31m74.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting semantic-version~=2.0 (from gradio>=4.0.0->llamafactory==0.7.2.dev0)\n",
            "  Downloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)\n",
            "Collecting tomlkit==0.12.0 (from gradio>=4.0.0->llamafactory==0.7.2.dev0)\n",
            "  Downloading tomlkit-0.12.0-py3-none-any.whl (37 kB)\n",
            "Collecting typer<1.0,>=0.12 (from gradio>=4.0.0->llamafactory==0.7.2.dev0)\n",
            "  Downloading typer-0.12.3-py3-none-any.whl (47 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m47.2/47.2 kB\u001b[0m \u001b[31m7.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: typing-extensions~=4.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=4.0.0->llamafactory==0.7.2.dev0) (4.12.0)\n",
            "Requirement already satisfied: urllib3~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=4.0.0->llamafactory==0.7.2.dev0) (2.0.7)\n",
            "Collecting websockets<12.0,>=10.0 (from gradio-client==0.17.0->gradio>=4.0.0->llamafactory==0.7.2.dev0)\n",
            "  Downloading websockets-11.0.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (129 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m129.9/129.9 kB\u001b[0m \u001b[31m20.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.7.0->llamafactory==0.7.2.dev0) (1.2.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.7.0->llamafactory==0.7.2.dev0) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.7.0->llamafactory==0.7.2.dev0) (4.52.4)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.7.0->llamafactory==0.7.2.dev0) (1.4.5)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.7.0->llamafactory==0.7.2.dev0) (3.1.2)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.7.0->llamafactory==0.7.2.dev0) (2.8.2)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic->llamafactory==0.7.2.dev0) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.18.3 in /usr/local/lib/python3.10/dist-packages (from pydantic->llamafactory==0.7.2.dev0) (2.18.3)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.1->llamafactory==0.7.2.dev0) (1.12.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.1->llamafactory==0.7.2.dev0) (3.3)\n",
            "Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch>=1.13.1->llamafactory==0.7.2.dev0)\n",
            "  Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)\n",
            "Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch>=1.13.1->llamafactory==0.7.2.dev0)\n",
            "  Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)\n",
            "Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch>=1.13.1->llamafactory==0.7.2.dev0)\n",
            "  Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)\n",
            "Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch>=1.13.1->llamafactory==0.7.2.dev0)\n",
            "  Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)\n",
            "Collecting nvidia-cublas-cu12==12.1.3.1 (from torch>=1.13.1->llamafactory==0.7.2.dev0)\n",
            "  Using cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)\n",
            "Collecting nvidia-cufft-cu12==11.0.2.54 (from torch>=1.13.1->llamafactory==0.7.2.dev0)\n",
            "  Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)\n",
            "Collecting nvidia-curand-cu12==10.3.2.106 (from torch>=1.13.1->llamafactory==0.7.2.dev0)\n",
            "  Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)\n",
            "Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch>=1.13.1->llamafactory==0.7.2.dev0)\n",
            "  Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)\n",
            "Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch>=1.13.1->llamafactory==0.7.2.dev0)\n",
            "  Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)\n",
            "Collecting nvidia-nccl-cu12==2.20.5 (from torch>=1.13.1->llamafactory==0.7.2.dev0)\n",
            "  Using cached nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl (176.2 MB)\n",
            "Collecting nvidia-nvtx-cu12==12.1.105 (from torch>=1.13.1->llamafactory==0.7.2.dev0)\n",
            "  Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)\n",
            "Requirement already satisfied: triton==2.3.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.1->llamafactory==0.7.2.dev0) (2.3.0)\n",
            "Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.13.1->llamafactory==0.7.2.dev0)\n",
            "  Downloading nvidia_nvjitlink_cu12-12.5.40-py3-none-manylinux2014_x86_64.whl (21.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.3/21.3 MB\u001b[0m \u001b[31m52.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.37.2->llamafactory==0.7.2.dev0) (2024.5.15)\n",
            "Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.37.2->llamafactory==0.7.2.dev0) (0.19.1)\n",
            "Collecting tyro>=0.5.11 (from trl>=0.8.1->llamafactory==0.7.2.dev0)\n",
            "  Downloading tyro-0.8.4-py3-none-any.whl (102 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m102.4/102.4 kB\u001b[0m \u001b[31m15.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/dist-packages (from uvicorn->llamafactory==0.7.2.dev0) (8.1.7)\n",
            "Collecting h11>=0.8 (from uvicorn->llamafactory==0.7.2.dev0)\n",
            "  Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m4.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting starlette<0.38.0,>=0.37.2 (from fastapi->llamafactory==0.7.2.dev0)\n",
            "  Downloading starlette-0.37.2-py3-none-any.whl (71 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m71.9/71.9 kB\u001b[0m \u001b[31m9.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting fastapi-cli>=0.0.2 (from fastapi->llamafactory==0.7.2.dev0)\n",
            "  Downloading fastapi_cli-0.0.4-py3-none-any.whl (9.5 kB)\n",
            "Collecting ujson!=4.0.2,!=4.1.0,!=4.2.0,!=4.3.0,!=5.0.0,!=5.1.0,>=4.0.1 (from fastapi->llamafactory==0.7.2.dev0)\n",
            "  Downloading ujson-5.10.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (53 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.6/53.6 kB\u001b[0m \u001b[31m8.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting email_validator>=2.0.0 (from fastapi->llamafactory==0.7.2.dev0)\n",
            "  Downloading email_validator-2.1.1-py3-none-any.whl (30 kB)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from fire->llamafactory==0.7.2.dev0) (1.16.0)\n",
            "Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from fire->llamafactory==0.7.2.dev0) (2.4.0)\n",
            "Requirement already satisfied: anyio in /usr/local/lib/python3.10/dist-packages (from sse-starlette->llamafactory==0.7.2.dev0) (3.7.1)\n",
            "Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio>=4.0.0->llamafactory==0.7.2.dev0) (0.4)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio>=4.0.0->llamafactory==0.7.2.dev0) (4.19.2)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio>=4.0.0->llamafactory==0.7.2.dev0) (0.12.1)\n",
            "Collecting dnspython>=2.0.0 (from email_validator>=2.0.0->fastapi->llamafactory==0.7.2.dev0)\n",
            "  Downloading dnspython-2.6.1-py3-none-any.whl (307 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m307.7/307.7 kB\u001b[0m \u001b[31m27.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: idna>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from email_validator>=2.0.0->fastapi->llamafactory==0.7.2.dev0) (3.7)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.14.3->llamafactory==0.7.2.dev0) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.14.3->llamafactory==0.7.2.dev0) (23.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.14.3->llamafactory==0.7.2.dev0) (1.4.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.14.3->llamafactory==0.7.2.dev0) (6.0.5)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.14.3->llamafactory==0.7.2.dev0) (1.9.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.14.3->llamafactory==0.7.2.dev0) (4.0.3)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx>=0.24.1->gradio>=4.0.0->llamafactory==0.7.2.dev0) (2024.2.2)\n",
            "Collecting httpcore==1.* (from httpx>=0.24.1->gradio>=4.0.0->llamafactory==0.7.2.dev0)\n",
            "  Downloading httpcore-1.0.5-py3-none-any.whl (77 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.9/77.9 kB\u001b[0m \u001b[31m8.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx>=0.24.1->gradio>=4.0.0->llamafactory==0.7.2.dev0) (1.3.1)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets>=2.14.3->llamafactory==0.7.2.dev0) (2023.4)\n",
            "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets>=2.14.3->llamafactory==0.7.2.dev0) (2024.1)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets>=2.14.3->llamafactory==0.7.2.dev0) (3.3.2)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio->sse-starlette->llamafactory==0.7.2.dev0) (1.2.1)\n",
            "Collecting shellingham>=1.3.0 (from typer<1.0,>=0.12->gradio>=4.0.0->llamafactory==0.7.2.dev0)\n",
            "  Downloading shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)\n",
            "Requirement already satisfied: rich>=10.11.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0,>=0.12->gradio>=4.0.0->llamafactory==0.7.2.dev0) (13.7.1)\n",
            "Requirement already satisfied: docstring-parser>=0.14.1 in /usr/local/lib/python3.10/dist-packages (from tyro>=0.5.11->trl>=0.8.1->llamafactory==0.7.2.dev0) (0.16)\n",
            "Collecting shtab>=1.5.6 (from tyro>=0.5.11->trl>=0.8.1->llamafactory==0.7.2.dev0)\n",
            "  Downloading shtab-1.7.1-py3-none-any.whl (14 kB)\n",
            "Collecting httptools>=0.5.0 (from uvicorn->llamafactory==0.7.2.dev0)\n",
            "  Downloading httptools-0.6.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (341 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m341.4/341.4 kB\u001b[0m \u001b[31m39.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting python-dotenv>=0.13 (from uvicorn->llamafactory==0.7.2.dev0)\n",
            "  Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)\n",
            "Collecting uvloop!=0.15.0,!=0.15.1,>=0.14.0 (from uvicorn->llamafactory==0.7.2.dev0)\n",
            "  Downloading uvloop-0.19.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.4/3.4 MB\u001b[0m \u001b[31m95.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting watchfiles>=0.13 (from uvicorn->llamafactory==0.7.2.dev0)\n",
            "  Downloading watchfiles-0.22.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m82.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: mpmath<1.4.0,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.13.1->llamafactory==0.7.2.dev0) (1.3.0)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio>=4.0.0->llamafactory==0.7.2.dev0) (2023.12.1)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio>=4.0.0->llamafactory==0.7.2.dev0) (0.35.1)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio>=4.0.0->llamafactory==0.7.2.dev0) (0.18.1)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.11.0->typer<1.0,>=0.12->gradio>=4.0.0->llamafactory==0.7.2.dev0) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.11.0->typer<1.0,>=0.12->gradio>=4.0.0->llamafactory==0.7.2.dev0) (2.16.1)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0,>=0.12->gradio>=4.0.0->llamafactory==0.7.2.dev0) (0.1.2)\n",
            "Building wheels for collected packages: fire, llamafactory, ffmpy\n",
            "  Building wheel for fire (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for fire: filename=fire-0.6.0-py2.py3-none-any.whl size=117029 sha256=91ce3c21bbb55f5e4c8ef182b7108e47756fcf41ef776e26cf964d67e73334b8\n",
            "  Stored in directory: /root/.cache/pip/wheels/d6/6d/5d/5b73fa0f46d01a793713f8859201361e9e581ced8c75e5c6a3\n",
            "  Building editable for llamafactory (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for llamafactory: filename=llamafactory-0.7.2.dev0-0.editable-py3-none-any.whl size=18708 sha256=713e86e5bdd8c9aa1b54ef20a43f857ca69c971664a1042418a54875313275b3\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-xr3igdxm/wheels/de/aa/c5/27b5682c5592b7c0eecc3e208f176dedf6b11a61cf2a910b85\n",
            "  Building wheel for ffmpy (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ffmpy: filename=ffmpy-0.3.2-py3-none-any.whl size=5584 sha256=520942341a1b687c010aed6923ec60a045dcf7c7b66b07b9dc62eb88216baa78\n",
            "  Stored in directory: /root/.cache/pip/wheels/bd/65/9a/671fc6dcde07d4418df0c592f8df512b26d7a0029c2a23dd81\n",
            "Successfully built fire llamafactory ffmpy\n",
            "Installing collected packages: pydub, ffmpy, xxhash, websockets, uvloop, ujson, tomlkit, shtab, shellingham, semantic-version, ruff, python-multipart, python-dotenv, orjson, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, httptools, h11, fire, einops, dnspython, dill, aiofiles, watchfiles, uvicorn, starlette, nvidia-cusparse-cu12, nvidia-cudnn-cu12, multiprocess, httpcore, email_validator, tyro, typer, sse-starlette, nvidia-cusolver-cu12, httpx, gradio-client, fastapi-cli, datasets, fastapi, bitsandbytes, accelerate, trl, peft, gradio, llamafactory\n",
            "  Attempting uninstall: typer\n",
            "    Found existing installation: typer 0.9.4\n",
            "    Uninstalling typer-0.9.4:\n",
            "      Successfully uninstalled typer-0.9.4\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "spacy 3.7.4 requires typer<0.10.0,>=0.3.0, but you have typer 0.12.3 which is incompatible.\n",
            "weasel 0.3.4 requires typer<0.10.0,>=0.3.0, but you have typer 0.12.3 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed accelerate-0.30.1 aiofiles-23.2.1 bitsandbytes-0.43.1 datasets-2.19.1 dill-0.3.8 dnspython-2.6.1 einops-0.8.0 email_validator-2.1.1 fastapi-0.111.0 fastapi-cli-0.0.4 ffmpy-0.3.2 fire-0.6.0 gradio-4.32.2 gradio-client-0.17.0 h11-0.14.0 httpcore-1.0.5 httptools-0.6.1 httpx-0.27.0 llamafactory-0.7.2.dev0 multiprocess-0.70.16 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.20.5 nvidia-nvjitlink-cu12-12.5.40 nvidia-nvtx-cu12-12.1.105 orjson-3.10.3 peft-0.11.1 pydub-0.25.1 python-dotenv-1.0.1 python-multipart-0.0.9 ruff-0.4.7 semantic-version-2.10.0 shellingham-1.5.4 shtab-1.7.1 sse-starlette-2.1.0 starlette-0.37.2 tomlkit-0.12.0 trl-0.8.6 typer-0.12.3 tyro-0.8.4 ujson-5.10.0 uvicorn-0.30.0 uvloop-0.19.0 watchfiles-0.22.0 websockets-11.0.3 xxhash-3.4.1\n"
          ]
        }
      ],
      "source": [
        "%cd /content/\n",
        "%rm -rf LLaMA-Factory\n",
        "!git clone https://github.com/hiyouga/LLaMA-Factory.git\n",
        "%cd LLaMA-Factory\n",
        "%ls\n",
        "!pip install -e .[torch,bitsandbytes]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Di9D2DY5dqmw"
      },
      "source": [
        "## Finetune Gemma\n",
        "\n",
        "開始 Gemma 2B 微調，使用這個 [demo Alpaca dataset](https://github.com/hiyouga/LLaMA-Factory/blob/main/data/alpaca_en_demo.json)。如果你想使用自己的數據集，請遵循這個 [guide from LLaMA Factory](https://github.com/hiyouga/LLaMA-Factory/tree/main/data)。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gWIzVxhwcDSw",
        "outputId": "4827e9bd-3bbb-444a-e05f-6e42008a6a56"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "/content/LLaMA-Factory\n",
            "2024-06-02 01:48:45.000610: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
            "2024-06-02 01:48:45.000673: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
            "2024-06-02 01:48:45.109651: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
            "2024-06-02 01:48:45.314212: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
            "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
            "2024-06-02 01:48:47.307591: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "06/02/2024 01:48:55 - WARNING - llamafactory.hparams.parser - We recommend enable `upcast_layernorm` in quantized training.\n",
            "06/02/2024 01:48:55 - INFO - llamafactory.hparams.parser - Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: False, compute dtype: torch.float16\n",
            "tokenizer_config.json: 100% 33.6k/33.6k [00:00<00:00, 45.7MB/s]\n",
            "tokenizer.model: 100% 4.24M/4.24M [00:00<00:00, 65.5MB/s]\n",
            "tokenizer.json: 100% 17.5M/17.5M [00:00<00:00, 76.5MB/s]\n",
            "special_tokens_map.json: 100% 636/636 [00:00<00:00, 4.21MB/s]\n",
            "[INFO|tokenization_utils_base.py:2108] 2024-06-02 01:48:56,428 >> loading file tokenizer.model from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/tokenizer.model\n",
            "[INFO|tokenization_utils_base.py:2108] 2024-06-02 01:48:56,428 >> loading file tokenizer.json from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/tokenizer.json\n",
            "[INFO|tokenization_utils_base.py:2108] 2024-06-02 01:48:56,428 >> loading file added_tokens.json from cache at None\n",
            "[INFO|tokenization_utils_base.py:2108] 2024-06-02 01:48:56,428 >> loading file special_tokens_map.json from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/special_tokens_map.json\n",
            "[INFO|tokenization_utils_base.py:2108] 2024-06-02 01:48:56,428 >> loading file tokenizer_config.json from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/tokenizer_config.json\n",
            "06/02/2024 01:48:57 - INFO - llamafactory.data.loader - Loading dataset alpaca_en_demo.json...\n",
            "Generating train split: 1000 examples [00:00, 34591.63 examples/s]\n",
            "Converting format of dataset: 100% 500/500 [00:00<00:00, 22261.81 examples/s]\n",
            "Running tokenizer on dataset: 100% 500/500 [00:00<00:00, 1159.80 examples/s]\n",
            "input_ids:\n",
            "[2, 106, 1645, 108, 81899, 1009, 2294, 2962, 8292, 577, 2582, 671, 31942, 235265, 108, 33552, 235290, 235274, 235315, 107, 108, 106, 2516, 108, 4858, 708, 1009, 11480, 2294, 2962, 8292, 577, 2582, 671, 31942, 576, 8789, 235290, 235274, 235315, 235292, 109, 235274, 235265, 7524, 50978, 235292, 99833, 1461, 577, 2745, 476, 235248, 235318, 235290, 48897, 6075, 774, 3588, 575, 2294, 6853, 578, 6858, 3387, 3764, 675, 1461, 1064, 708, 11666, 235265, 109, 235284, 235265, 95061, 27503, 235292, 2009, 13521, 1461, 577, 8044, 27503, 2183, 575, 2294, 6853, 577, 7704, 573, 8151, 576, 573, 12369, 235265, 109, 235304, 235265, 6442, 40039, 235292, 99833, 1461, 577, 9903, 1024, 4961, 12283, 675, 19386, 578, 2003, 604, 235248, 235284, 235276, 10102, 235269, 689, 1281, 1634, 106470, 675, 696, 3476, 235248, 235318, 235276, 235358, 12678, 235265, 109, 235310, 235265, 10262, 53959, 235292, 41038, 578, 53316, 9278, 1064, 791, 2063, 1280, 3387, 3764, 675, 28757, 9108, 235265, 109, 235308, 235265, 22126, 235292, 35369, 573, 17315, 576, 8603, 9674, 577, 21422, 573, 8566, 576, 573, 12369, 235265, 109, 235318, 235265, 161610, 235292, 50803, 49586, 8292, 604, 9278, 1064, 791, 12272, 6222, 604, 573, 12369, 689, 791, 2063, 1280, 3764, 675, 28757, 9108, 235265, 109, 235324, 235265, 159171, 235292, 99833, 1461, 577, 947, 48878, 1185, 40553, 3831, 22153, 2506, 235265, 109, 235321, 235265, 5665, 17512, 235292, 230867, 573, 2294, 1105, 573, 12369, 235269, 1277, 24023, 235269, 578, 1368, 577, 4692, 9606, 1593, 124719, 235269, 151767, 235269, 2294, 2566, 68170, 235269, 578, 3127, 4562, 235265, 109, 235315, 235265, 99281, 578, 5562, 235292, 32481, 573, 1758, 578, 2395, 576, 71828, 578, 5562, 235265, 109, 235274, 235276, 235265, 14892, 16842, 235292, 50803, 5056, 16842, 577, 4089, 573, 8151, 576, 573, 12369, 1865, 11707, 578, 5605, 235265, 1]\n",
            "inputs:\n",
            "<bos><start_of_turn>user\n",
            "Suggest some public health measures to control an outbreak.\n",
            "COVID-19<end_of_turn>\n",
            "<start_of_turn>model\n",
            "Here are some suggested public health measures to control an outbreak of COVID-19:\n",
            "\n",
            "1. Social distancing: Encourage people to keep a 6-feet distance from others in public places and avoid close contact with people who are sick.\n",
            "\n",
            "2. Wearing masks: Advise people to wear masks while in public places to prevent the spread of the virus.\n",
            "\n",
            "3. Hand hygiene: Encourage people to wash their hands frequently with soap and water for 20 seconds, or use hand sanitizer with at least 60% alcohol.\n",
            "\n",
            "4. Contact tracing: Identify and isolate individuals who have come into close contact with infected persons.\n",
            "\n",
            "5. Testing: Increase the availability of testing facilities to detect the presence of the virus.\n",
            "\n",
            "6. Quarantine: Implement quarantine measures for individuals who have tested positive for the virus or have come into contact with infected persons.\n",
            "\n",
            "7. Vaccines: Encourage people to get vaccinated when vaccines become readily available.\n",
            "\n",
            "8. Public awareness: Educate the public about the virus, its prevention, and how to stay healthy through flyers, billboards, public service announcements, and social media.\n",
            "\n",
            "9. Gathering and events: Limit the number and size of gatherings and events.\n",
            "\n",
            "10. Travel restrictions: Implement travel restrictions to limit the spread of the virus between communities and countries.<eos>\n",
            "label_ids:\n",
            "[-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 4858, 708, 1009, 11480, 2294, 2962, 8292, 577, 2582, 671, 31942, 576, 8789, 235290, 235274, 235315, 235292, 109, 235274, 235265, 7524, 50978, 235292, 99833, 1461, 577, 2745, 476, 235248, 235318, 235290, 48897, 6075, 774, 3588, 575, 2294, 6853, 578, 6858, 3387, 3764, 675, 1461, 1064, 708, 11666, 235265, 109, 235284, 235265, 95061, 27503, 235292, 2009, 13521, 1461, 577, 8044, 27503, 2183, 575, 2294, 6853, 577, 7704, 573, 8151, 576, 573, 12369, 235265, 109, 235304, 235265, 6442, 40039, 235292, 99833, 1461, 577, 9903, 1024, 4961, 12283, 675, 19386, 578, 2003, 604, 235248, 235284, 235276, 10102, 235269, 689, 1281, 1634, 106470, 675, 696, 3476, 235248, 235318, 235276, 235358, 12678, 235265, 109, 235310, 235265, 10262, 53959, 235292, 41038, 578, 53316, 9278, 1064, 791, 2063, 1280, 3387, 3764, 675, 28757, 9108, 235265, 109, 235308, 235265, 22126, 235292, 35369, 573, 17315, 576, 8603, 9674, 577, 21422, 573, 8566, 576, 573, 12369, 235265, 109, 235318, 235265, 161610, 235292, 50803, 49586, 8292, 604, 9278, 1064, 791, 12272, 6222, 604, 573, 12369, 689, 791, 2063, 1280, 3764, 675, 28757, 9108, 235265, 109, 235324, 235265, 159171, 235292, 99833, 1461, 577, 947, 48878, 1185, 40553, 3831, 22153, 2506, 235265, 109, 235321, 235265, 5665, 17512, 235292, 230867, 573, 2294, 1105, 573, 12369, 235269, 1277, 24023, 235269, 578, 1368, 577, 4692, 9606, 1593, 124719, 235269, 151767, 235269, 2294, 2566, 68170, 235269, 578, 3127, 4562, 235265, 109, 235315, 235265, 99281, 578, 5562, 235292, 32481, 573, 1758, 578, 2395, 576, 71828, 578, 5562, 235265, 109, 235274, 235276, 235265, 14892, 16842, 235292, 50803, 5056, 16842, 577, 4089, 573, 8151, 576, 573, 12369, 1865, 11707, 578, 5605, 235265, 1]\n",
            "labels:\n",
            "Here are some suggested public health measures to control an outbreak of COVID-19:\n",
            "\n",
            "1. Social distancing: Encourage people to keep a 6-feet distance from others in public places and avoid close contact with people who are sick.\n",
            "\n",
            "2. Wearing masks: Advise people to wear masks while in public places to prevent the spread of the virus.\n",
            "\n",
            "3. Hand hygiene: Encourage people to wash their hands frequently with soap and water for 20 seconds, or use hand sanitizer with at least 60% alcohol.\n",
            "\n",
            "4. Contact tracing: Identify and isolate individuals who have come into close contact with infected persons.\n",
            "\n",
            "5. Testing: Increase the availability of testing facilities to detect the presence of the virus.\n",
            "\n",
            "6. Quarantine: Implement quarantine measures for individuals who have tested positive for the virus or have come into contact with infected persons.\n",
            "\n",
            "7. Vaccines: Encourage people to get vaccinated when vaccines become readily available.\n",
            "\n",
            "8. Public awareness: Educate the public about the virus, its prevention, and how to stay healthy through flyers, billboards, public service announcements, and social media.\n",
            "\n",
            "9. Gathering and events: Limit the number and size of gatherings and events.\n",
            "\n",
            "10. Travel restrictions: Implement travel restrictions to limit the spread of the virus between communities and countries.<eos>\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "config.json: 100% 627/627 [00:00<00:00, 4.03MB/s]\n",
            "[INFO|configuration_utils.py:733] 2024-06-02 01:48:58,499 >> loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/config.json\n",
            "[INFO|configuration_utils.py:796] 2024-06-02 01:48:58,502 >> Model config GemmaConfig {\n",
            "  \"_name_or_path\": \"google/gemma-2b\",\n",
            "  \"architectures\": [\n",
            "    \"GemmaForCausalLM\"\n",
            "  ],\n",
            "  \"attention_bias\": false,\n",
            "  \"attention_dropout\": 0.0,\n",
            "  \"bos_token_id\": 2,\n",
            "  \"eos_token_id\": 1,\n",
            "  \"head_dim\": 256,\n",
            "  \"hidden_act\": \"gelu\",\n",
            "  \"hidden_activation\": null,\n",
            "  \"hidden_size\": 2048,\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"intermediate_size\": 16384,\n",
            "  \"max_position_embeddings\": 8192,\n",
            "  \"model_type\": \"gemma\",\n",
            "  \"num_attention_heads\": 8,\n",
            "  \"num_hidden_layers\": 18,\n",
            "  \"num_key_value_heads\": 1,\n",
            "  \"pad_token_id\": 0,\n",
            "  \"rms_norm_eps\": 1e-06,\n",
            "  \"rope_scaling\": null,\n",
            "  \"rope_theta\": 10000.0,\n",
            "  \"torch_dtype\": \"bfloat16\",\n",
            "  \"transformers_version\": \"4.41.1\",\n",
            "  \"use_cache\": true,\n",
            "  \"vocab_size\": 256000\n",
            "}\n",
            "\n",
            "06/02/2024 01:48:58 - INFO - llamafactory.model.utils.quantization - Quantizing model to 4 bit.\n",
            "model.safetensors.index.json: 100% 13.5k/13.5k [00:00<00:00, 49.9MB/s]\n",
            "[INFO|modeling_utils.py:3474] 2024-06-02 01:48:58,733 >> loading weights file model.safetensors from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/model.safetensors.index.json\n",
            "Downloading shards:   0% 0/2 [00:00<?, ?it/s]\n",
            "model-00001-of-00002.safetensors:   0% 0.00/4.95G [00:00<?, ?B/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:   1% 31.5M/4.95G [00:00<00:21, 229MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:   1% 62.9M/4.95G [00:00<00:20, 240MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:   2% 94.4M/4.95G [00:00<00:23, 206MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:   3% 126M/4.95G [00:00<00:28, 169MB/s] \u001b[A\n",
            "model-00001-of-00002.safetensors:   3% 147M/4.95G [00:00<00:28, 168MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:   3% 168M/4.95G [00:00<00:27, 172MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:   4% 189M/4.95G [00:01<00:26, 178MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:   4% 210M/4.95G [00:01<00:26, 180MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:   5% 231M/4.95G [00:01<00:25, 186MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:   5% 262M/4.95G [00:01<00:23, 202MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:   6% 294M/4.95G [00:01<00:21, 213MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:   7% 325M/4.95G [00:01<00:20, 223MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:   7% 357M/4.95G [00:01<00:21, 210MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:   8% 388M/4.95G [00:01<00:23, 198MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:   8% 409M/4.95G [00:02<00:23, 196MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:   9% 440M/4.95G [00:02<00:22, 201MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:   9% 461M/4.95G [00:02<00:22, 200MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  10% 493M/4.95G [00:02<00:21, 204MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  10% 514M/4.95G [00:02<00:22, 195MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  11% 535M/4.95G [00:02<00:22, 196MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  11% 566M/4.95G [00:02<00:21, 200MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  12% 587M/4.95G [00:02<00:22, 196MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  13% 619M/4.95G [00:03<00:22, 193MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  13% 640M/4.95G [00:03<00:23, 180MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  13% 661M/4.95G [00:03<00:25, 165MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  14% 682M/4.95G [00:03<00:25, 167MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  14% 703M/4.95G [00:03<00:25, 166MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  15% 724M/4.95G [00:04<01:29, 47.1MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  15% 744M/4.95G [00:05<01:37, 43.1MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  15% 755M/4.95G [00:07<03:55, 17.8MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  16% 786M/4.95G [00:07<02:21, 29.3MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  16% 807M/4.95G [00:08<01:52, 36.8MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  17% 839M/4.95G [00:08<01:15, 54.3MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  17% 860M/4.95G [00:08<01:00, 67.5MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  18% 881M/4.95G [00:08<00:49, 82.1MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  18% 902M/4.95G [00:08<00:41, 98.2MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  19% 923M/4.95G [00:08<00:34, 115MB/s] \u001b[A\n",
            "model-00001-of-00002.safetensors:  19% 944M/4.95G [00:08<00:30, 132MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  20% 975M/4.95G [00:08<00:25, 158MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  20% 1.01G/4.95G [00:09<00:22, 176MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  21% 1.03G/4.95G [00:09<00:27, 141MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  21% 1.05G/4.95G [00:09<00:30, 128MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  22% 1.07G/4.95G [00:09<00:29, 133MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  22% 1.09G/4.95G [00:09<00:26, 143MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  22% 1.11G/4.95G [00:09<00:24, 157MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  23% 1.13G/4.95G [00:09<00:23, 165MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  24% 1.16G/4.95G [00:10<00:22, 169MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  24% 1.18G/4.95G [00:10<00:21, 175MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  24% 1.21G/4.95G [00:11<00:54, 69.2MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  25% 1.23G/4.95G [00:11<00:53, 69.1MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  25% 1.25G/4.95G [00:11<00:48, 76.0MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  26% 1.27G/4.95G [00:11<00:43, 83.6MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  26% 1.29G/4.95G [00:12<00:44, 82.4MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  26% 1.30G/4.95G [00:12<00:42, 84.8MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  27% 1.33G/4.95G [00:12<00:36, 97.9MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  27% 1.35G/4.95G [00:12<00:39, 90.5MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  28% 1.36G/4.95G [00:12<00:51, 69.9MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  28% 1.38G/4.95G [00:13<00:42, 83.3MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  28% 1.41G/4.95G [00:13<00:37, 95.6MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  29% 1.43G/4.95G [00:13<00:32, 108MB/s] \u001b[A\n",
            "model-00001-of-00002.safetensors:  29% 1.45G/4.95G [00:13<00:28, 123MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  30% 1.47G/4.95G [00:13<00:30, 113MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  30% 1.49G/4.95G [00:13<00:30, 113MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  31% 1.52G/4.95G [00:14<00:23, 143MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  31% 1.55G/4.95G [00:14<00:20, 165MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  32% 1.57G/4.95G [00:14<00:20, 168MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  32% 1.59G/4.95G [00:14<00:19, 169MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  33% 1.61G/4.95G [00:14<00:19, 175MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  33% 1.65G/4.95G [00:14<00:17, 188MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  34% 1.67G/4.95G [00:14<00:17, 192MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  34% 1.69G/4.95G [00:16<01:04, 50.4MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  35% 1.71G/4.95G [00:17<02:04, 25.9MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  35% 1.74G/4.95G [00:17<01:22, 39.0MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  36% 1.77G/4.95G [00:18<00:57, 54.9MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  36% 1.80G/4.95G [00:18<00:42, 73.3MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  37% 1.82G/4.95G [00:18<00:36, 86.5MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  37% 1.85G/4.95G [00:18<00:31, 97.2MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  38% 1.87G/4.95G [00:18<00:28, 108MB/s] \u001b[A\n",
            "model-00001-of-00002.safetensors:  38% 1.90G/4.95G [00:18<00:22, 133MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  39% 1.93G/4.95G [00:18<00:19, 153MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  40% 1.96G/4.95G [00:19<00:17, 171MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  40% 1.99G/4.95G [00:19<00:15, 185MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  41% 2.02G/4.95G [00:19<00:14, 195MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  42% 2.06G/4.95G [00:19<00:14, 193MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  42% 2.08G/4.95G [00:19<00:14, 193MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  42% 2.10G/4.95G [00:19<00:14, 191MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  43% 2.12G/4.95G [00:19<00:14, 191MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  43% 2.14G/4.95G [00:19<00:14, 195MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  44% 2.16G/4.95G [00:20<00:14, 196MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  44% 2.18G/4.95G [00:21<00:47, 58.0MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  45% 2.20G/4.95G [00:21<00:37, 72.9MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  45% 2.22G/4.95G [00:21<00:30, 89.1MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  45% 2.24G/4.95G [00:21<00:25, 105MB/s] \u001b[A\n",
            "model-00001-of-00002.safetensors:  46% 2.26G/4.95G [00:21<00:21, 122MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  46% 2.29G/4.95G [00:21<00:19, 138MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  47% 2.31G/4.95G [00:21<00:17, 151MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  47% 2.33G/4.95G [00:21<00:16, 160MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  47% 2.35G/4.95G [00:21<00:15, 168MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  48% 2.37G/4.95G [00:22<00:15, 167MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  48% 2.39G/4.95G [00:22<00:15, 169MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  49% 2.41G/4.95G [00:22<00:14, 175MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  49% 2.43G/4.95G [00:22<00:14, 173MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  50% 2.45G/4.95G [00:22<00:13, 178MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  50% 2.47G/4.95G [00:22<00:14, 172MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  50% 2.50G/4.95G [00:22<00:13, 179MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  51% 2.52G/4.95G [00:23<00:19, 123MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  52% 2.55G/4.95G [00:23<00:15, 150MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  52% 2.57G/4.95G [00:23<00:14, 161MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  52% 2.59G/4.95G [00:23<00:13, 171MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  53% 2.61G/4.95G [00:23<00:14, 158MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  53% 2.63G/4.95G [00:23<00:15, 154MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  54% 2.66G/4.95G [00:23<00:13, 175MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  54% 2.68G/4.95G [00:24<00:16, 136MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  55% 2.71G/4.95G [00:24<00:14, 150MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  55% 2.73G/4.95G [00:24<00:15, 145MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  56% 2.75G/4.95G [00:24<00:24, 88.2MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  56% 2.77G/4.95G [00:25<00:35, 62.2MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  56% 2.78G/4.95G [00:25<00:32, 66.8MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  57% 2.80G/4.95G [00:25<00:27, 78.6MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  57% 2.82G/4.95G [00:25<00:25, 81.9MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  57% 2.83G/4.95G [00:26<00:25, 84.2MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  57% 2.84G/4.95G [00:26<00:24, 85.6MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  58% 2.85G/4.95G [00:26<00:25, 82.9MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  58% 2.87G/4.95G [00:26<00:22, 92.9MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  59% 2.89G/4.95G [00:26<00:19, 107MB/s] \u001b[A\n",
            "model-00001-of-00002.safetensors:  59% 2.92G/4.95G [00:26<00:17, 119MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  59% 2.94G/4.95G [00:26<00:16, 123MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  60% 2.96G/4.95G [00:27<00:18, 107MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  60% 2.98G/4.95G [00:27<00:18, 106MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  61% 3.00G/4.95G [00:27<00:18, 106MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  61% 3.02G/4.95G [00:27<00:17, 109MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  61% 3.04G/4.95G [00:27<00:17, 112MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  62% 3.06G/4.95G [00:28<00:15, 122MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  62% 3.08G/4.95G [00:28<00:13, 136MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  63% 3.10G/4.95G [00:28<00:12, 145MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  63% 3.12G/4.95G [00:28<00:12, 149MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  64% 3.15G/4.95G [00:28<00:12, 149MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  64% 3.17G/4.95G [00:28<00:11, 156MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  64% 3.19G/4.95G [00:28<00:10, 162MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  65% 3.21G/4.95G [00:28<00:10, 164MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  66% 3.24G/4.95G [00:29<00:09, 177MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  66% 3.26G/4.95G [00:29<00:09, 182MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  66% 3.28G/4.95G [00:29<00:09, 184MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  67% 3.30G/4.95G [00:29<00:08, 187MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  67% 3.32G/4.95G [00:29<00:09, 167MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  68% 3.34G/4.95G [00:29<00:09, 168MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  68% 3.37G/4.95G [00:29<00:09, 167MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  68% 3.39G/4.95G [00:29<00:09, 170MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  69% 3.41G/4.95G [00:30<00:09, 167MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  69% 3.43G/4.95G [00:32<00:52, 28.8MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  70% 3.46G/4.95G [00:32<00:33, 43.9MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  71% 3.49G/4.95G [00:32<00:23, 62.1MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  71% 3.52G/4.95G [00:32<00:17, 82.9MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  72% 3.54G/4.95G [00:32<00:14, 97.2MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  72% 3.57G/4.95G [00:32<00:12, 110MB/s] \u001b[A\n",
            "model-00001-of-00002.safetensors:  73% 3.59G/4.95G [00:32<00:11, 122MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  73% 3.61G/4.95G [00:33<00:09, 138MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  73% 3.63G/4.95G [00:33<00:08, 153MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  74% 3.66G/4.95G [00:33<00:07, 173MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  74% 3.68G/4.95G [00:33<00:07, 179MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  75% 3.70G/4.95G [00:33<00:06, 183MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  75% 3.73G/4.95G [00:33<00:06, 194MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  76% 3.75G/4.95G [00:33<00:06, 171MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  76% 3.77G/4.95G [00:33<00:07, 165MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  77% 3.80G/4.95G [00:34<00:09, 126MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  77% 3.82G/4.95G [00:34<00:08, 135MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  78% 3.85G/4.95G [00:34<00:06, 158MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  78% 3.87G/4.95G [00:35<00:23, 46.2MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  79% 3.89G/4.95G [00:35<00:18, 55.9MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  79% 3.91G/4.95G [00:36<00:14, 69.8MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  80% 3.94G/4.95G [00:36<00:10, 94.9MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  80% 3.97G/4.95G [00:36<00:08, 120MB/s] \u001b[A\n",
            "model-00001-of-00002.safetensors:  81% 4.01G/4.95G [00:36<00:06, 141MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  81% 4.03G/4.95G [00:36<00:06, 148MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  82% 4.05G/4.95G [00:37<00:19, 45.5MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  82% 4.07G/4.95G [00:38<00:16, 52.7MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  83% 4.09G/4.95G [00:38<00:13, 62.4MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  83% 4.11G/4.95G [00:38<00:11, 72.2MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  84% 4.14G/4.95G [00:38<00:08, 91.6MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  84% 4.16G/4.95G [00:38<00:07, 107MB/s] \u001b[A\n",
            "model-00001-of-00002.safetensors:  85% 4.18G/4.95G [00:38<00:06, 122MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  85% 4.20G/4.95G [00:40<00:17, 43.3MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  85% 4.23G/4.95G [00:40<00:13, 52.8MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  86% 4.25G/4.95G [00:40<00:13, 51.9MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  86% 4.27G/4.95G [00:41<00:10, 63.5MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  87% 4.29G/4.95G [00:41<00:08, 77.0MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  87% 4.31G/4.95G [00:41<00:06, 91.8MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  88% 4.33G/4.95G [00:41<00:05, 106MB/s] \u001b[A\n",
            "model-00001-of-00002.safetensors:  88% 4.35G/4.95G [00:41<00:04, 119MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  88% 4.37G/4.95G [00:41<00:04, 130MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  89% 4.39G/4.95G [00:41<00:04, 137MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  89% 4.41G/4.95G [00:41<00:03, 146MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  90% 4.44G/4.95G [00:42<00:03, 147MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  90% 4.46G/4.95G [00:42<00:03, 151MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  91% 4.48G/4.95G [00:42<00:03, 138MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  91% 4.50G/4.95G [00:42<00:03, 146MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  92% 4.53G/4.95G [00:42<00:02, 166MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  92% 4.56G/4.95G [00:42<00:02, 181MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  93% 4.58G/4.95G [00:42<00:01, 182MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  93% 4.60G/4.95G [00:43<00:01, 172MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  94% 4.62G/4.95G [00:43<00:01, 169MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  94% 4.65G/4.95G [00:43<00:01, 161MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  94% 4.67G/4.95G [00:43<00:01, 166MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  95% 4.69G/4.95G [00:43<00:01, 171MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  95% 4.71G/4.95G [00:43<00:01, 160MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  96% 4.73G/4.95G [00:43<00:01, 155MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  96% 4.75G/4.95G [00:44<00:02, 94.8MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  96% 4.77G/4.95G [00:44<00:01, 103MB/s] \u001b[A\n",
            "model-00001-of-00002.safetensors:  97% 4.79G/4.95G [00:44<00:01, 108MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  97% 4.81G/4.95G [00:44<00:01, 122MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  98% 4.84G/4.95G [00:44<00:00, 146MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  98% 4.87G/4.95G [00:45<00:01, 55.8MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  99% 4.89G/4.95G [00:45<00:00, 70.1MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors:  99% 4.91G/4.95G [00:46<00:00, 86.5MB/s]\u001b[A\n",
            "model-00001-of-00002.safetensors: 100% 4.95G/4.95G [00:46<00:00, 107MB/s]\n",
            "Downloading shards:  50% 1/2 [00:46<00:46, 46.32s/it]\n",
            "model-00002-of-00002.safetensors:   0% 0.00/67.1M [00:00<?, ?B/s]\u001b[A\n",
            "model-00002-of-00002.safetensors:  31% 21.0M/67.1M [00:00<00:00, 197MB/s]\u001b[A\n",
            "model-00002-of-00002.safetensors:  62% 41.9M/67.1M [00:00<00:00, 197MB/s]\u001b[A\n",
            "model-00002-of-00002.safetensors: 100% 67.1M/67.1M [00:00<00:00, 190MB/s]\n",
            "Downloading shards: 100% 2/2 [00:46<00:00, 23.37s/it]\n",
            "[INFO|modeling_utils.py:1519] 2024-06-02 01:49:45,468 >> Instantiating GemmaForCausalLM model under default dtype torch.float16.\n",
            "[INFO|configuration_utils.py:962] 2024-06-02 01:49:45,470 >> Generate config GenerationConfig {\n",
            "  \"bos_token_id\": 2,\n",
            "  \"eos_token_id\": 1,\n",
            "  \"pad_token_id\": 0\n",
            "}\n",
            "\n",
            "[WARNING|logging.py:329] 2024-06-02 01:49:45,566 >> `config.hidden_act` is ignored, you should use `config.hidden_activation` instead.\n",
            "Gemma's activation function will be set to `gelu_pytorch_tanh`. Please, use\n",
            "`config.hidden_activation` if you want to override this behaviour.\n",
            "See https://github.com/huggingface/transformers/pull/29402 for more details.\n",
            "Loading checkpoint shards: 100% 2/2 [00:23<00:00, 11.77s/it]\n",
            "[INFO|modeling_utils.py:4280] 2024-06-02 01:50:10,023 >> All model checkpoint weights were used when initializing GemmaForCausalLM.\n",
            "\n",
            "[INFO|modeling_utils.py:4288] 2024-06-02 01:50:10,023 >> All the weights of GemmaForCausalLM were initialized from the model checkpoint at google/gemma-2b.\n",
            "If your task is similar to the task the model of the checkpoint was trained on, you can already use GemmaForCausalLM for predictions without further training.\n",
            "generation_config.json: 100% 137/137 [00:00<00:00, 655kB/s]\n",
            "[INFO|configuration_utils.py:917] 2024-06-02 01:50:10,089 >> loading configuration file generation_config.json from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/generation_config.json\n",
            "[INFO|configuration_utils.py:962] 2024-06-02 01:50:10,090 >> Generate config GenerationConfig {\n",
            "  \"bos_token_id\": 2,\n",
            "  \"eos_token_id\": 1,\n",
            "  \"pad_token_id\": 0\n",
            "}\n",
            "\n",
            "06/02/2024 01:50:10 - INFO - llamafactory.model.utils.checkpointing - Gradient checkpointing enabled.\n",
            "06/02/2024 01:50:10 - INFO - llamafactory.model.utils.attention - Using torch SDPA for faster training and inference.\n",
            "06/02/2024 01:50:10 - INFO - llamafactory.model.adapter - Upcasting trainable params to float32.\n",
            "06/02/2024 01:50:10 - INFO - llamafactory.model.adapter - Fine-tuning method: LoRA\n",
            "06/02/2024 01:50:10 - INFO - llamafactory.model.utils.misc - Found linear modules: v_proj,k_proj,o_proj,gate_proj,down_proj,up_proj,q_proj\n",
            "06/02/2024 01:50:10 - INFO - llamafactory.model.loader - trainable params: 9805824 || all params: 2515978240 || trainable%: 0.3897\n",
            "[INFO|trainer.py:641] 2024-06-02 01:50:10,612 >> Using auto half precision backend\n",
            "06/02/2024 01:50:11 - INFO - llamafactory.train.utils - Using LoRA+ optimizer with loraplus lr ratio 16.00.\n",
            "[INFO|trainer.py:2078] 2024-06-02 01:50:11,023 >> ***** Running training *****\n",
            "[INFO|trainer.py:2079] 2024-06-02 01:50:11,023 >>   Num examples = 500\n",
            "[INFO|trainer.py:2080] 2024-06-02 01:50:11,023 >>   Num Epochs = 3\n",
            "[INFO|trainer.py:2081] 2024-06-02 01:50:11,023 >>   Instantaneous batch size per device = 2\n",
            "[INFO|trainer.py:2084] 2024-06-02 01:50:11,023 >>   Total train batch size (w. parallel, distributed & accumulation) = 8\n",
            "[INFO|trainer.py:2085] 2024-06-02 01:50:11,023 >>   Gradient Accumulation steps = 4\n",
            "[INFO|trainer.py:2086] 2024-06-02 01:50:11,023 >>   Total optimization steps = 186\n",
            "[INFO|trainer.py:2087] 2024-06-02 01:50:11,027 >>   Number of trainable parameters = 9,805,824\n",
            "{'loss': 1.4483, 'grad_norm': 0.4991081655025482, 'learning_rate': 2.6315789473684212e-05, 'epoch': 0.16}\n",
            "{'loss': 1.3751, 'grad_norm': 0.5427204370498657, 'learning_rate': 4.999557652060729e-05, 'epoch': 0.32}\n",
            "{'loss': 1.2121, 'grad_norm': 0.5160215497016907, 'learning_rate': 4.946665048328287e-05, 'epoch': 0.48}\n",
            "{'loss': 1.208, 'grad_norm': 0.5302833914756775, 'learning_rate': 4.807442755497524e-05, 'epoch': 0.64}\n",
            "{'loss': 1.2563, 'grad_norm': 1.2021236419677734, 'learning_rate': 4.586803181690609e-05, 'epoch': 0.8}\n",
            "{'loss': 1.2072, 'grad_norm': 0.7239830493927002, 'learning_rate': 4.292531514268008e-05, 'epoch': 0.96}\n",
            "{'loss': 1.083, 'grad_norm': 0.7575322985649109, 'learning_rate': 3.9350110223152844e-05, 'epoch': 1.12}\n",
            "{'loss': 1.0284, 'grad_norm': 0.8508509993553162, 'learning_rate': 3.526856686758269e-05, 'epoch': 1.28}\n",
            "{'loss': 0.9012, 'grad_norm': 0.7356946468353271, 'learning_rate': 3.082470085335133e-05, 'epoch': 1.44}\n",
            "{'loss': 0.934, 'grad_norm': 0.735954999923706, 'learning_rate': 2.6175312381477442e-05, 'epoch': 1.6}\n",
            "{'loss': 0.9584, 'grad_norm': 0.9395563006401062, 'learning_rate': 2.148445343837755e-05, 'epoch': 1.76}\n",
            "{'loss': 0.9926, 'grad_norm': 1.1644067764282227, 'learning_rate': 1.69176392810087e-05, 'epoch': 1.92}\n",
            "{'loss': 0.8619, 'grad_norm': 0.9992528557777405, 'learning_rate': 1.2636008291040618e-05, 'epoch': 2.08}\n",
            "{'loss': 0.8154, 'grad_norm': 0.7435272336006165, 'learning_rate': 8.790636265485334e-06, 'epoch': 2.24}\n",
            "{'loss': 0.8083, 'grad_norm': 1.05929434299469, 'learning_rate': 5.51720576197794e-06, 'epoch': 2.4}\n",
            "{'loss': 0.7884, 'grad_norm': 0.7872556447982788, 'learning_rate': 2.931218588927315e-06, 'epoch': 2.56}\n",
            "{'loss': 0.672, 'grad_norm': 0.7596755623817444, 'learning_rate': 1.1239203660860648e-06, 'epoch': 2.72}\n",
            "{'loss': 0.6431, 'grad_norm': 0.5819358825683594, 'learning_rate': 1.5908095594207583e-07, 'epoch': 2.88}\n",
            "100% 186/186 [08:28<00:00,  3.01s/it][INFO|trainer.py:2329] 2024-06-02 01:58:40,017 >> \n",
            "\n",
            "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
            "\n",
            "\n",
            "{'train_runtime': 508.9906, 'train_samples_per_second': 2.947, 'train_steps_per_second': 0.365, 'train_loss': 1.0038108261682654, 'epoch': 2.98}\n",
            "100% 186/186 [08:28<00:00,  2.74s/it]\n",
            "[INFO|trainer.py:3410] 2024-06-02 01:58:40,020 >> Saving model checkpoint to gemma_lora\n",
            "[INFO|configuration_utils.py:733] 2024-06-02 01:58:40,262 >> loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/config.json\n",
            "[INFO|configuration_utils.py:796] 2024-06-02 01:58:40,263 >> Model config GemmaConfig {\n",
            "  \"architectures\": [\n",
            "    \"GemmaForCausalLM\"\n",
            "  ],\n",
            "  \"attention_bias\": false,\n",
            "  \"attention_dropout\": 0.0,\n",
            "  \"bos_token_id\": 2,\n",
            "  \"eos_token_id\": 1,\n",
            "  \"head_dim\": 256,\n",
            "  \"hidden_act\": \"gelu\",\n",
            "  \"hidden_activation\": null,\n",
            "  \"hidden_size\": 2048,\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"intermediate_size\": 16384,\n",
            "  \"max_position_embeddings\": 8192,\n",
            "  \"model_type\": \"gemma\",\n",
            "  \"num_attention_heads\": 8,\n",
            "  \"num_hidden_layers\": 18,\n",
            "  \"num_key_value_heads\": 1,\n",
            "  \"pad_token_id\": 0,\n",
            "  \"rms_norm_eps\": 1e-06,\n",
            "  \"rope_scaling\": null,\n",
            "  \"rope_theta\": 10000.0,\n",
            "  \"torch_dtype\": \"bfloat16\",\n",
            "  \"transformers_version\": \"4.41.1\",\n",
            "  \"use_cache\": true,\n",
            "  \"vocab_size\": 256000\n",
            "}\n",
            "\n",
            "[INFO|tokenization_utils_base.py:2513] 2024-06-02 01:58:40,448 >> tokenizer config file saved in gemma_lora/tokenizer_config.json\n",
            "[INFO|tokenization_utils_base.py:2522] 2024-06-02 01:58:40,453 >> Special tokens file saved in gemma_lora/special_tokens_map.json\n",
            "***** train metrics *****\n",
            "  epoch                    =      2.976\n",
            "  total_flos               =  4188229GF\n",
            "  train_loss               =     1.0038\n",
            "  train_runtime            = 0:08:28.99\n",
            "  train_samples_per_second =      2.947\n",
            "  train_steps_per_second   =      0.365\n",
            "[INFO|modelcard.py:450] 2024-06-02 01:58:40,965 >> Dropping the following result as it does not have all the necessary fields:\n",
            "{'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}}\n"
          ]
        }
      ],
      "source": [
        "import json\n",
        "\n",
        "args = dict(\n",
        "    stage=\"sft\",  # do supervised fine-tuning\n",
        "    do_train=True,\n",
        "    model_name_or_path=\"google/gemma-2b\",  # use bnb-4bit-quantized Gemma 2B model\n",
        "    dataset=\"alpaca_en_demo\",  # use the demo alpaca datasets\n",
        "    template=\"gemma\",  # use Gemma prompt template\n",
        "    finetuning_type=\"lora\",  # use LoRA adapters to save memory\n",
        "    lora_target=\"all\",  # attach LoRA adapters to all linear layers\n",
        "    output_dir=\"gemma_lora\",  # the path to save LoRA adapters\n",
        "    per_device_train_batch_size=2,  # the batch size\n",
        "    gradient_accumulation_steps=4,  # the gradient accumulation steps\n",
        "    lr_scheduler_type=\"cosine\",  # use cosine learning rate scheduler\n",
        "    logging_steps=10,  # log every 10 steps\n",
        "    warmup_ratio=0.1,  # use warmup scheduler\n",
        "    save_steps=1000,  # save checkpoint every 1000 steps\n",
        "    learning_rate=5e-5,  # the learning rate\n",
        "    num_train_epochs=3.0,  # the epochs of training\n",
        "    max_samples=500,  # use 500 examples in each dataset\n",
        "    max_grad_norm=1.0,  # clip gradient norm to 1.0\n",
        "    quantization_bit=4,  # use 4-bit QLoRA\n",
        "    loraplus_lr_ratio=16.0,  # use LoRA+ algorithm with lambda=16.0\n",
        "    fp16=True,  # use float16 mixed precision training\n",
        ")\n",
        "\n",
        "json.dump(args, open(\"train_gemma.json\", \"w\", encoding=\"utf-8\"), indent=2)\n",
        "\n",
        "%cd /content/LLaMA-Factory/\n",
        "\n",
        "!llamafactory-cli train train_gemma.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6hJbdNtZrANr"
      },
      "source": [
        "## 在聊天設定中執行推論"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000,
          "referenced_widgets": [
            "3d1b6ab8ca8e4bb5978fe2b32d9f09bc",
            "35b7447f99714f8093d0def80702b8ce",
            "0da44d3e21044e4c9ea38a0ec42d5a0d",
            "39404874553f41aab27788d96e122f35",
            "8cf8781120ab40b09de84b78b9e38726",
            "53d852152fe742baa7dd81f5af020796",
            "f984f9e1d31e4c07813bf44ef5c61925",
            "b9df7df513fe4810addfbefae98dc4b9",
            "3da14b51edc64eb3b1e5a1d47e438fcc",
            "bdd56d50debc4060b26cfcf17b7c638a",
            "14e07cff44704835a0a7bf2bd09df670"
          ]
        },
        "id": "2pGX3hLubhkJ",
        "outputId": "37817eed-0e7e-4f61-8e7b-0189053c93df"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "/content/LLaMA-Factory/src\n",
            "/content/LLaMA-Factory\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "[INFO|tokenization_utils_base.py:2108] 2024-06-02 01:59:02,909 >> loading file tokenizer.model from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/tokenizer.model\n",
            "[INFO|tokenization_utils_base.py:2108] 2024-06-02 01:59:02,910 >> loading file tokenizer.json from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/tokenizer.json\n",
            "[INFO|tokenization_utils_base.py:2108] 2024-06-02 01:59:02,912 >> loading file added_tokens.json from cache at None\n",
            "[INFO|tokenization_utils_base.py:2108] 2024-06-02 01:59:02,914 >> loading file special_tokens_map.json from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/special_tokens_map.json\n",
            "[INFO|tokenization_utils_base.py:2108] 2024-06-02 01:59:02,916 >> loading file tokenizer_config.json from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/tokenizer_config.json\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "[INFO|configuration_utils.py:733] 2024-06-02 01:59:04,168 >> loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/config.json\n",
            "[INFO|configuration_utils.py:796] 2024-06-02 01:59:04,173 >> Model config GemmaConfig {\n",
            "  \"_name_or_path\": \"google/gemma-2b\",\n",
            "  \"architectures\": [\n",
            "    \"GemmaForCausalLM\"\n",
            "  ],\n",
            "  \"attention_bias\": false,\n",
            "  \"attention_dropout\": 0.0,\n",
            "  \"bos_token_id\": 2,\n",
            "  \"eos_token_id\": 1,\n",
            "  \"head_dim\": 256,\n",
            "  \"hidden_act\": \"gelu\",\n",
            "  \"hidden_activation\": null,\n",
            "  \"hidden_size\": 2048,\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"intermediate_size\": 16384,\n",
            "  \"max_position_embeddings\": 8192,\n",
            "  \"model_type\": \"gemma\",\n",
            "  \"num_attention_heads\": 8,\n",
            "  \"num_hidden_layers\": 18,\n",
            "  \"num_key_value_heads\": 1,\n",
            "  \"pad_token_id\": 0,\n",
            "  \"rms_norm_eps\": 1e-06,\n",
            "  \"rope_scaling\": null,\n",
            "  \"rope_theta\": 10000.0,\n",
            "  \"torch_dtype\": \"bfloat16\",\n",
            "  \"transformers_version\": \"4.41.1\",\n",
            "  \"use_cache\": true,\n",
            "  \"vocab_size\": 256000\n",
            "}\n",
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "06/02/2024 01:59:04 - INFO - llamafactory.model.utils.quantization - Quantizing model to 4 bit.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:llamafactory.model.utils.quantization:Quantizing model to 4 bit.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "06/02/2024 01:59:04 - INFO - llamafactory.model.patcher - Using KV cache for faster generation.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:llamafactory.model.patcher:Using KV cache for faster generation.\n",
            "[INFO|modeling_utils.py:3474] 2024-06-02 01:59:04,316 >> loading weights file model.safetensors from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/model.safetensors.index.json\n",
            "[INFO|modeling_utils.py:1519] 2024-06-02 01:59:04,322 >> Instantiating GemmaForCausalLM model under default dtype torch.bfloat16.\n",
            "[INFO|configuration_utils.py:962] 2024-06-02 01:59:04,324 >> Generate config GenerationConfig {\n",
            "  \"bos_token_id\": 2,\n",
            "  \"eos_token_id\": 1,\n",
            "  \"pad_token_id\": 0\n",
            "}\n",
            "\n",
            "[WARNING|logging.py:329] 2024-06-02 01:59:04,333 >> `config.hidden_act` is ignored, you should use `config.hidden_activation` instead.\n",
            "Gemma's activation function will be set to `gelu_pytorch_tanh`. Please, use\n",
            "`config.hidden_activation` if you want to override this behaviour.\n",
            "See https://github.com/huggingface/transformers/pull/29402 for more details.\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "3d1b6ab8ca8e4bb5978fe2b32d9f09bc",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "[INFO|modeling_utils.py:4280] 2024-06-02 01:59:10,951 >> All model checkpoint weights were used when initializing GemmaForCausalLM.\n",
            "\n",
            "[INFO|modeling_utils.py:4288] 2024-06-02 01:59:10,956 >> All the weights of GemmaForCausalLM were initialized from the model checkpoint at google/gemma-2b.\n",
            "If your task is similar to the task the model of the checkpoint was trained on, you can already use GemmaForCausalLM for predictions without further training.\n",
            "[INFO|configuration_utils.py:917] 2024-06-02 01:59:10,993 >> loading configuration file generation_config.json from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/generation_config.json\n",
            "[INFO|configuration_utils.py:962] 2024-06-02 01:59:10,995 >> Generate config GenerationConfig {\n",
            "  \"bos_token_id\": 2,\n",
            "  \"eos_token_id\": 1,\n",
            "  \"pad_token_id\": 0\n",
            "}\n",
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "06/02/2024 01:59:11 - INFO - llamafactory.model.utils.attention - Using torch SDPA for faster training and inference.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:llamafactory.model.utils.attention:Using torch SDPA for faster training and inference.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "06/02/2024 01:59:11 - INFO - llamafactory.model.adapter - Upcasting trainable params to float32.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:llamafactory.model.adapter:Upcasting trainable params to float32.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "06/02/2024 01:59:11 - INFO - llamafactory.model.adapter - Fine-tuning method: LoRA\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:llamafactory.model.adapter:Fine-tuning method: LoRA\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "06/02/2024 01:59:11 - INFO - llamafactory.model.adapter - Loaded adapter(s): gemma_lora\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:llamafactory.model.adapter:Loaded adapter(s): gemma_lora\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "06/02/2024 01:59:11 - INFO - llamafactory.model.loader - all params: 2515978240\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:llamafactory.model.loader:all params: 2515978240\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Welcome to the CLI application, use `clear` to remove the history, use `exit` to exit the application.\n",
            "\n",
            "User: where is Chicago?\n",
            "Assistant: Chicago is located in the U.S. state of Illinois, and is the third most populous city in the United States.\n",
            "\n",
            "User: exit\n"
          ]
        }
      ],
      "source": [
        "%cd /content/LLaMA-Factory/src/\n",
        "\n",
        "from llamafactory.chat import ChatModel\n",
        "from llamafactory.extras.misc import torch_gc\n",
        "\n",
        "%cd /content/LLaMA-Factory/\n",
        "\n",
        "args = dict(\n",
        "    model_name_or_path=\"google/gemma-2b\",  # use Gemma 2B model\n",
        "    adapter_name_or_path=\"gemma_lora\",  # load the saved LoRA adapters\n",
        "    template=\"gemma\",  # same to the one in training\n",
        "    finetuning_type=\"lora\",  # same to the one in training\n",
        "    quantization_bit=4,  # load 4-bit quantized model\n",
        ")\n",
        "chat_model = ChatModel(args)\n",
        "\n",
        "messages = []\n",
        "print(\n",
        "    \"Welcome to the CLI application, use `clear` to remove the history, use `exit` to exit the application.\"\n",
        ")\n",
        "while True:\n",
        "    query = input(\"\\nUser: \")\n",
        "    if query.strip() == \"exit\":\n",
        "        break\n",
        "    if query.strip() == \"clear\":\n",
        "        messages = []\n",
        "        torch_gc()\n",
        "        print(\"History has been removed.\")\n",
        "        continue\n",
        "\n",
        "    messages.append({\"role\": \"user\", \"content\": query})\n",
        "    print(\"Assistant: \", end=\"\", flush=True)\n",
        "\n",
        "    response = \"\"\n",
        "    for new_text in chat_model.stream_chat(messages):\n",
        "        print(new_text, end=\"\", flush=True)\n",
        "        response += new_text\n",
        "    print()\n",
        "    messages.append({\"role\": \"assistant\", \"content\": response})\n",
        "\n",
        "torch_gc()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oeZUSRbHhbV2"
      },
      "source": [
        "## 合併 LoRA 適配器並上傳微調後的模型到 Hugging Face"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w84le7s5jyY_",
        "outputId": "45a34588-4046-4a44-8f89-ba5a4d871c17"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "/content/LLaMA-Factory\n",
            "2024-06-02 01:59:36.861478: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
            "2024-06-02 01:59:36.861538: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
            "2024-06-02 01:59:36.862938: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
            "2024-06-02 01:59:38.404919: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "[INFO|tokenization_utils_base.py:2108] 2024-06-02 01:59:47,841 >> loading file tokenizer.model from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/tokenizer.model\n",
            "[INFO|tokenization_utils_base.py:2108] 2024-06-02 01:59:47,842 >> loading file tokenizer.json from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/tokenizer.json\n",
            "[INFO|tokenization_utils_base.py:2108] 2024-06-02 01:59:47,842 >> loading file added_tokens.json from cache at None\n",
            "[INFO|tokenization_utils_base.py:2108] 2024-06-02 01:59:47,842 >> loading file special_tokens_map.json from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/special_tokens_map.json\n",
            "[INFO|tokenization_utils_base.py:2108] 2024-06-02 01:59:47,842 >> loading file tokenizer_config.json from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/tokenizer_config.json\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "[INFO|configuration_utils.py:733] 2024-06-02 01:59:49,029 >> loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/config.json\n",
            "[INFO|configuration_utils.py:796] 2024-06-02 01:59:49,031 >> Model config GemmaConfig {\n",
            "  \"_name_or_path\": \"google/gemma-2b\",\n",
            "  \"architectures\": [\n",
            "    \"GemmaForCausalLM\"\n",
            "  ],\n",
            "  \"attention_bias\": false,\n",
            "  \"attention_dropout\": 0.0,\n",
            "  \"bos_token_id\": 2,\n",
            "  \"eos_token_id\": 1,\n",
            "  \"head_dim\": 256,\n",
            "  \"hidden_act\": \"gelu\",\n",
            "  \"hidden_activation\": null,\n",
            "  \"hidden_size\": 2048,\n",
            "  \"initializer_range\": 0.02,\n",
            "  \"intermediate_size\": 16384,\n",
            "  \"max_position_embeddings\": 8192,\n",
            "  \"model_type\": \"gemma\",\n",
            "  \"num_attention_heads\": 8,\n",
            "  \"num_hidden_layers\": 18,\n",
            "  \"num_key_value_heads\": 1,\n",
            "  \"pad_token_id\": 0,\n",
            "  \"rms_norm_eps\": 1e-06,\n",
            "  \"rope_scaling\": null,\n",
            "  \"rope_theta\": 10000.0,\n",
            "  \"torch_dtype\": \"bfloat16\",\n",
            "  \"transformers_version\": \"4.41.1\",\n",
            "  \"use_cache\": true,\n",
            "  \"vocab_size\": 256000\n",
            "}\n",
            "\n",
            "06/02/2024 01:59:49 - INFO - llamafactory.model.patcher - Using KV cache for faster generation.\n",
            "[INFO|modeling_utils.py:3474] 2024-06-02 01:59:49,150 >> loading weights file model.safetensors from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/model.safetensors.index.json\n",
            "[INFO|modeling_utils.py:1519] 2024-06-02 01:59:49,152 >> Instantiating GemmaForCausalLM model under default dtype torch.bfloat16.\n",
            "[INFO|configuration_utils.py:962] 2024-06-02 01:59:49,153 >> Generate config GenerationConfig {\n",
            "  \"bos_token_id\": 2,\n",
            "  \"eos_token_id\": 1,\n",
            "  \"pad_token_id\": 0\n",
            "}\n",
            "\n",
            "[WARNING|logging.py:329] 2024-06-02 01:59:49,156 >> `config.hidden_act` is ignored, you should use `config.hidden_activation` instead.\n",
            "Gemma's activation function will be set to `gelu_pytorch_tanh`. Please, use\n",
            "`config.hidden_activation` if you want to override this behaviour.\n",
            "See https://github.com/huggingface/transformers/pull/29402 for more details.\n",
            "Loading checkpoint shards: 100% 2/2 [00:00<00:00,  2.65it/s]\n",
            "[INFO|modeling_utils.py:4280] 2024-06-02 01:59:49,958 >> All model checkpoint weights were used when initializing GemmaForCausalLM.\n",
            "\n",
            "[INFO|modeling_utils.py:4288] 2024-06-02 01:59:49,959 >> All the weights of GemmaForCausalLM were initialized from the model checkpoint at google/gemma-2b.\n",
            "If your task is similar to the task the model of the checkpoint was trained on, you can already use GemmaForCausalLM for predictions without further training.\n",
            "[INFO|configuration_utils.py:917] 2024-06-02 01:59:49,987 >> loading configuration file generation_config.json from cache at /root/.cache/huggingface/hub/models--google--gemma-2b/snapshots/2ac59a5d7bf4e1425010f0d457dde7d146658953/generation_config.json\n",
            "[INFO|configuration_utils.py:962] 2024-06-02 01:59:49,987 >> Generate config GenerationConfig {\n",
            "  \"bos_token_id\": 2,\n",
            "  \"eos_token_id\": 1,\n",
            "  \"pad_token_id\": 0\n",
            "}\n",
            "\n",
            "06/02/2024 01:59:49 - INFO - llamafactory.model.utils.attention - Using torch SDPA for faster training and inference.\n",
            "06/02/2024 01:59:49 - INFO - llamafactory.model.adapter - Upcasting trainable params to float32.\n",
            "06/02/2024 01:59:49 - INFO - llamafactory.model.adapter - Fine-tuning method: LoRA\n",
            "06/02/2024 02:00:42 - INFO - llamafactory.model.adapter - Merged 1 adapter(s).\n",
            "06/02/2024 02:00:42 - INFO - llamafactory.model.adapter - Loaded adapter(s): gemma_lora\n",
            "06/02/2024 02:00:42 - INFO - llamafactory.model.loader - all params: 2506172416\n",
            "[INFO|configuration_utils.py:472] 2024-06-02 02:00:42,108 >> Configuration saved in gemma_lora_merged/config.json\n",
            "[INFO|configuration_utils.py:731] 2024-06-02 02:00:42,108 >> Configuration saved in gemma_lora_merged/generation_config.json\n",
            "[INFO|modeling_utils.py:2626] 2024-06-02 02:01:32,681 >> The model is bigger than the maximum size per checkpoint (2GB) and is going to be split in 3 checkpoint shards. You can find where each parameters has been saved in the index located at gemma_lora_merged/model.safetensors.index.json.\n",
            "[INFO|configuration_utils.py:472] 2024-06-02 02:01:33,029 >> Configuration saved in /tmp/tmpumli7anw/config.json\n",
            "[INFO|configuration_utils.py:731] 2024-06-02 02:01:33,030 >> Configuration saved in /tmp/tmpumli7anw/generation_config.json\n",
            "[INFO|modeling_utils.py:2626] 2024-06-02 02:06:35,330 >> The model is bigger than the maximum size per checkpoint (2GB) and is going to be split in 3 checkpoint shards. You can find where each parameters has been saved in the index located at /tmp/tmpumli7anw/model.safetensors.index.json.\n",
            "[INFO|hub.py:759] 2024-06-02 02:07:02,840 >> Uploading the following files to windmaple/gemma-2b-finetuned-model-llama-factory: generation_config.json,config.json,model-00003-of-00003.safetensors,model-00001-of-00003.safetensors,model-00002-of-00003.safetensors,README.md,model.safetensors.index.json\n",
            "model-00003-of-00003.safetensors:   0% 0.00/1.08G [00:00<?, ?B/s]\n",
            "model-00001-of-00003.safetensors:   0% 0.00/1.95G [00:00<?, ?B/s]\u001b[A\n",
            "\n",
            "Upload 3 LFS files:   0% 0/3 [00:00<?, ?it/s]\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:   0% 16.4k/1.08G [00:00<2:53:28, 104kB/s]\n",
            "model-00001-of-00003.safetensors:   0% 16.4k/1.95G [00:00<5:46:27, 93.7kB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:   1% 6.72M/1.08G [00:00<00:33, 31.7MB/s] \n",
            "model-00001-of-00003.safetensors:   0% 2.87M/1.95G [00:00<02:31, 12.9MB/s]  \u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:   1% 15.9M/1.08G [00:00<00:19, 56.0MB/s]\n",
            "model-00001-of-00003.safetensors:   0% 6.14M/1.95G [00:00<01:34, 20.6MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:   0% 4.75M/1.98G [00:00<02:11, 15.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:   1% 13.4M/1.95G [00:00<00:49, 39.0MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:   1% 11.0M/1.98G [00:00<01:05, 30.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:   1% 16.0M/1.98G [00:00<00:56, 34.5MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:   1% 17.7M/1.95G [00:00<01:12, 26.8MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:   2% 30.7M/1.98G [00:00<00:30, 64.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:   2% 22.2M/1.08G [00:00<00:45, 23.2MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:   2% 26.5M/1.08G [00:01<00:43, 24.5MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:   2% 43.0M/1.98G [00:01<00:37, 52.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:   2% 32.0M/1.95G [00:01<01:04, 29.9MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:   2% 43.1M/1.95G [00:01<00:41, 45.5MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:   3% 32.0M/1.08G [00:01<00:52, 20.2MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:   4% 40.5M/1.08G [00:01<00:34, 29.8MB/s]\n",
            "model-00001-of-00003.safetensors:   3% 49.3M/1.95G [00:01<00:51, 36.6MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:   3% 57.4M/1.95G [00:01<00:42, 44.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:   4% 47.3M/1.08G [00:01<00:30, 33.9MB/s]\n",
            "model-00001-of-00003.safetensors:   3% 63.4M/1.95G [00:01<00:42, 44.7MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:   5% 52.1M/1.08G [00:01<00:35, 29.2MB/s]\n",
            "model-00001-of-00003.safetensors:   4% 68.8M/1.95G [00:01<00:49, 37.7MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:   6% 61.1M/1.08G [00:01<00:26, 39.1MB/s]\n",
            "model-00001-of-00003.safetensors:   4% 74.3M/1.95G [00:02<00:47, 39.5MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:   4% 87.8M/1.98G [00:02<00:46, 40.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:   6% 66.3M/1.08G [00:02<00:31, 32.5MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:   7% 74.7M/1.08G [00:02<00:24, 41.8MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:   5% 99.9M/1.98G [00:02<00:46, 40.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:   4% 84.5M/1.95G [00:02<00:56, 32.7MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:   5% 104M/1.98G [00:02<00:46, 40.3MB/s] \u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:   5% 91.7M/1.95G [00:02<00:48, 38.2MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:   6% 111M/1.98G [00:02<00:43, 43.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:   5% 96.1M/1.95G [00:02<00:58, 31.6MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:   6% 115M/1.98G [00:02<00:49, 37.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:   5% 107M/1.95G [00:02<00:39, 46.3MB/s] \u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:   8% 89.0M/1.08G [00:03<00:34, 28.9MB/s]\n",
            "model-00001-of-00003.safetensors:   6% 113M/1.95G [00:03<00:47, 38.3MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:   9% 95.9M/1.08G [00:03<00:28, 34.8MB/s]\n",
            "model-00001-of-00003.safetensors:   6% 120M/1.95G [00:03<00:41, 44.3MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:   7% 134M/1.98G [00:03<00:45, 40.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:   6% 125M/1.95G [00:03<00:41, 44.0MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  11% 116M/1.08G [00:03<00:27, 34.7MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:   7% 144M/1.98G [00:03<01:15, 24.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:   8% 152M/1.98G [00:03<00:56, 32.5MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  12% 126M/1.08G [00:03<00:21, 44.3MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:   8% 157M/1.98G [00:04<00:51, 35.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:   7% 134M/1.95G [00:04<01:18, 23.1MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:   7% 140M/1.95G [00:04<01:02, 29.1MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  12% 132M/1.08G [00:04<00:25, 36.9MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  13% 137M/1.08G [00:04<00:24, 38.6MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:   9% 173M/1.98G [00:04<00:43, 41.8MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  13% 144M/1.08G [00:04<00:22, 42.4MB/s]\n",
            "model-00003-of-00003.safetensors:  14% 149M/1.08G [00:04<00:24, 37.4MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  14% 155M/1.08G [00:04<00:23, 40.1MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:   9% 183M/1.98G [00:04<00:48, 37.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:   8% 160M/1.95G [00:04<00:56, 31.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  15% 160M/1.08G [00:04<00:25, 36.0MB/s]\n",
            "model-00003-of-00003.safetensors:  15% 167M/1.08G [00:04<00:21, 42.4MB/s]\n",
            "model-00001-of-00003.safetensors:   9% 174M/1.95G [00:04<00:41, 43.2MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  16% 173M/1.08G [00:05<00:20, 44.7MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  10% 200M/1.98G [00:05<00:46, 38.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  18% 190M/1.08G [00:05<00:16, 52.5MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  11% 210M/1.98G [00:05<00:56, 31.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  19% 206M/1.08G [00:05<00:14, 59.4MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  20% 213M/1.08G [00:05<00:19, 44.3MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  21% 224M/1.08G [00:06<00:19, 43.1MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  22% 233M/1.08G [00:06<00:17, 49.7MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  13% 258M/1.98G [00:06<00:36, 47.8MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  23% 254M/1.08G [00:06<00:13, 60.7MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  24% 262M/1.08G [00:06<00:15, 53.4MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  25% 272M/1.08G [00:07<00:14, 54.8MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  26% 281M/1.08G [00:07<00:13, 59.8MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  27% 288M/1.08G [00:07<00:16, 49.4MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  28% 299M/1.08G [00:07<00:12, 60.8MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  29% 319M/1.08G [00:07<00:11, 64.8MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  17% 336M/1.98G [00:07<00:33, 48.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  18% 350M/1.98G [00:07<00:25, 64.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  18% 358M/1.98G [00:08<00:27, 58.8MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  19% 368M/1.98G [00:08<00:29, 54.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  19% 383M/1.98G [00:08<00:23, 69.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  20% 391M/1.98G [00:08<00:37, 41.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  20% 400M/1.98G [00:09<00:39, 40.5MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  30% 326M/1.08G [00:09<00:45, 16.5MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  32% 347M/1.08G [00:09<00:26, 27.8MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  22% 432M/1.98G [00:09<00:32, 48.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  34% 367M/1.08G [00:09<00:17, 41.7MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  23% 453M/1.98G [00:10<00:33, 46.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  35% 375M/1.08G [00:10<00:19, 36.7MB/s]\n",
            "model-00001-of-00003.safetensors:   9% 183M/1.95G [00:10<06:29, 4.53MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  35% 384M/1.08G [00:10<00:19, 36.6MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  36% 392M/1.08G [00:10<00:16, 41.9MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  37% 400M/1.08G [00:10<00:14, 46.3MB/s]\n",
            "model-00001-of-00003.safetensors:  10% 192M/1.95G [00:10<04:25, 6.62MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  24% 485M/1.98G [00:10<00:35, 41.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  37% 406M/1.08G [00:10<00:17, 37.6MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  25% 490M/1.98G [00:10<00:34, 43.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  11% 208M/1.95G [00:11<02:14, 12.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  38% 411M/1.08G [00:11<00:17, 38.8MB/s]\n",
            "model-00001-of-00003.safetensors:  11% 213M/1.95G [00:11<02:00, 14.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  38% 416M/1.08G [00:11<00:19, 33.7MB/s]\n",
            "model-00001-of-00003.safetensors:  11% 220M/1.95G [00:11<01:26, 20.0MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  39% 422M/1.08G [00:11<00:17, 38.7MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  40% 428M/1.08G [00:11<00:16, 39.8MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  40% 432M/1.08G [00:11<00:18, 35.5MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  27% 528M/1.98G [00:11<00:25, 56.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  41% 440M/1.08G [00:11<00:14, 43.3MB/s]\n",
            "model-00003-of-00003.safetensors:  41% 446M/1.08G [00:11<00:13, 48.0MB/s]\n",
            "model-00001-of-00003.safetensors:  12% 237M/1.95G [00:11<01:08, 24.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  27% 534M/1.98G [00:11<00:33, 43.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  42% 451M/1.08G [00:12<00:16, 38.0MB/s]\n",
            "model-00003-of-00003.safetensors:  42% 457M/1.08G [00:12<00:14, 42.3MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  27% 544M/1.98G [00:12<00:38, 37.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  43% 464M/1.08G [00:12<00:12, 47.6MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  28% 550M/1.98G [00:12<00:35, 40.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  13% 254M/1.95G [00:12<00:51, 32.8MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  28% 556M/1.98G [00:12<00:33, 43.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  43% 469M/1.08G [00:12<00:20, 30.3MB/s]\n",
            "model-00001-of-00003.safetensors:  14% 267M/1.95G [00:12<00:39, 42.3MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  44% 476M/1.08G [00:12<00:17, 35.0MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  29% 566M/1.98G [00:12<00:37, 38.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  29% 572M/1.98G [00:12<00:33, 41.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  44% 481M/1.08G [00:12<00:18, 32.8MB/s]\n",
            "model-00003-of-00003.safetensors:  45% 488M/1.08G [00:13<00:15, 39.1MB/s]\n",
            "model-00003-of-00003.safetensors:  46% 494M/1.08G [00:13<00:13, 44.5MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  29% 576M/1.98G [00:13<00:46, 30.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  29% 584M/1.98G [00:13<00:34, 40.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  46% 499M/1.08G [00:13<00:17, 33.3MB/s]\n",
            "model-00003-of-00003.safetensors:  47% 512M/1.08G [00:13<00:12, 44.9MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  30% 592M/1.98G [00:13<00:45, 30.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  31% 606M/1.98G [00:13<00:28, 48.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  16% 304M/1.95G [00:13<00:53, 30.5MB/s]\u001b[A\n",
            "model-00003-of-00003.safetensors:  48% 517M/1.08G [00:13<00:18, 30.3MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  48% 522M/1.08G [00:14<00:16, 33.5MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  31% 619M/1.98G [00:14<00:30, 44.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  49% 527M/1.08G [00:14<00:15, 36.1MB/s]\n",
            "model-00003-of-00003.safetensors:  49% 532M/1.08G [00:14<00:17, 32.2MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  50% 540M/1.08G [00:14<00:12, 42.7MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  32% 632M/1.98G [00:14<00:34, 39.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  17% 337M/1.95G [00:14<00:45, 35.6MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  32% 638M/1.98G [00:14<00:32, 41.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  50% 545M/1.08G [00:14<00:15, 34.5MB/s]\n",
            "model-00001-of-00003.safetensors:  18% 352M/1.95G [00:14<00:35, 44.7MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  51% 552M/1.08G [00:14<00:13, 38.6MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  52% 558M/1.08G [00:14<00:12, 42.3MB/s]\n",
            "model-00001-of-00003.safetensors:  18% 357M/1.95G [00:15<00:42, 37.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  33% 654M/1.98G [00:15<00:30, 44.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  52% 563M/1.08G [00:15<00:15, 33.6MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  53% 569M/1.08G [00:15<00:13, 38.4MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  34% 666M/1.98G [00:15<00:29, 44.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  53% 575M/1.08G [00:15<00:11, 43.2MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  34% 672M/1.98G [00:15<00:30, 43.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  19% 374M/1.95G [00:15<00:41, 38.2MB/s]\u001b[A\n",
            "model-00003-of-00003.safetensors:  54% 580M/1.08G [00:15<00:15, 33.1MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  34% 676M/1.98G [00:15<00:35, 37.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  54% 586M/1.08G [00:15<00:12, 38.4MB/s]\n",
            "model-00001-of-00003.safetensors:  20% 389M/1.95G [00:15<00:41, 37.5MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  55% 592M/1.08G [00:15<00:12, 39.5MB/s]\n",
            "model-00001-of-00003.safetensors:  20% 394M/1.95G [00:15<00:40, 38.3MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  55% 596M/1.08G [00:16<00:14, 32.6MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  56% 602M/1.08G [00:16<00:12, 37.1MB/s]\n",
            "model-00001-of-00003.safetensors:  21% 400M/1.95G [00:16<00:48, 31.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  36% 705M/1.98G [00:16<00:33, 38.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  56% 608M/1.08G [00:16<00:14, 32.6MB/s]\n",
            "model-00001-of-00003.safetensors:  21% 414M/1.95G [00:16<00:35, 43.0MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  36% 712M/1.98G [00:16<00:29, 43.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  57% 616M/1.08G [00:16<00:12, 37.6MB/s]\n",
            "model-00003-of-00003.safetensors:  57% 622M/1.08G [00:16<00:11, 41.0MB/s]\n",
            "model-00001-of-00003.safetensors:  22% 424M/1.95G [00:16<00:39, 38.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  58% 626M/1.08G [00:16<00:13, 33.3MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  37% 731M/1.98G [00:16<00:29, 42.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  58% 632M/1.08G [00:17<00:11, 38.7MB/s]\n",
            "model-00003-of-00003.safetensors:  59% 639M/1.08G [00:17<00:10, 43.8MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  37% 736M/1.98G [00:17<00:38, 32.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  23% 445M/1.95G [00:17<00:33, 45.2MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  60% 650M/1.08G [00:17<00:10, 40.0MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  38% 752M/1.98G [00:17<00:29, 41.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  23% 450M/1.95G [00:17<00:49, 30.2MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  61% 656M/1.08G [00:17<00:11, 35.6MB/s]\n",
            "model-00001-of-00003.safetensors:  23% 456M/1.95G [00:17<00:42, 34.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  39% 767M/1.98G [00:17<00:23, 52.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  62% 672M/1.08G [00:18<00:12, 33.9MB/s]\n",
            "model-00001-of-00003.safetensors:  24% 467M/1.95G [00:18<00:55, 26.6MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  63% 679M/1.08G [00:18<00:10, 39.2MB/s]\n",
            "model-00001-of-00003.safetensors:  24% 473M/1.95G [00:18<00:45, 32.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  63% 683M/1.08G [00:18<00:09, 41.2MB/s]\n",
            "model-00001-of-00003.safetensors:  25% 478M/1.95G [00:18<00:40, 35.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  40% 783M/1.98G [00:18<00:33, 35.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  64% 688M/1.08G [00:18<00:12, 31.4MB/s]\n",
            "model-00001-of-00003.safetensors:  25% 483M/1.95G [00:18<00:53, 27.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  64% 694M/1.08G [00:18<00:10, 36.8MB/s]\n",
            "model-00001-of-00003.safetensors:  25% 488M/1.95G [00:18<00:44, 33.1MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  65% 699M/1.08G [00:18<00:09, 39.2MB/s]\n",
            "model-00003-of-00003.safetensors:  65% 704M/1.08G [00:18<00:08, 42.8MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  41% 805M/1.98G [00:18<00:33, 35.5MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  26% 497M/1.95G [00:19<00:49, 29.3MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  65% 709M/1.08G [00:19<00:10, 34.3MB/s]\n",
            "model-00003-of-00003.safetensors:  66% 714M/1.08G [00:19<00:09, 37.8MB/s]\n",
            "model-00003-of-00003.safetensors:  66% 719M/1.08G [00:19<00:09, 40.0MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  41% 821M/1.98G [00:19<00:32, 35.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  26% 514M/1.95G [00:19<00:42, 33.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  67% 723M/1.08G [00:19<00:11, 31.7MB/s]\n",
            "model-00003-of-00003.safetensors:  67% 730M/1.08G [00:19<00:09, 38.3MB/s]\n",
            "model-00001-of-00003.safetensors:  27% 526M/1.95G [00:19<00:33, 42.6MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  68% 736M/1.08G [00:19<00:08, 40.8MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  42% 842M/1.98G [00:19<00:28, 39.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  68% 740M/1.08G [00:19<00:10, 33.9MB/s]\n",
            "model-00003-of-00003.safetensors:  69% 752M/1.08G [00:20<00:08, 37.9MB/s]\n",
            "model-00001-of-00003.safetensors:  28% 544M/1.95G [00:20<00:41, 34.0MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  70% 761M/1.08G [00:20<00:06, 49.8MB/s]\n",
            "model-00003-of-00003.safetensors:  71% 767M/1.08G [00:20<00:06, 51.3MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  43% 855M/1.98G [00:20<00:38, 29.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  29% 558M/1.95G [00:20<00:32, 42.7MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  72% 783M/1.08G [00:20<00:05, 55.2MB/s]\n",
            "model-00001-of-00003.safetensors:  29% 563M/1.95G [00:20<00:44, 31.5MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  44% 865M/1.98G [00:20<00:42, 26.5MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  29% 573M/1.95G [00:20<00:30, 44.8MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  44% 873M/1.98G [00:20<00:31, 34.8MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  73% 789M/1.08G [00:21<00:08, 34.3MB/s]\n",
            "model-00001-of-00003.safetensors:  30% 587M/1.95G [00:21<00:27, 48.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  73% 795M/1.08G [00:21<00:07, 37.6MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  45% 888M/1.98G [00:21<00:28, 38.8MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  74% 800M/1.08G [00:21<00:08, 33.8MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  45% 893M/1.98G [00:21<00:26, 41.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  74% 806M/1.08G [00:21<00:07, 37.9MB/s]\n",
            "model-00003-of-00003.safetensors:  75% 813M/1.08G [00:21<00:06, 44.1MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  45% 898M/1.98G [00:21<00:32, 33.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  46% 904M/1.98G [00:21<00:27, 39.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  76% 818M/1.08G [00:21<00:07, 37.2MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  46% 912M/1.98G [00:21<00:22, 47.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  76% 824M/1.08G [00:21<00:06, 40.7MB/s]\n",
            "model-00003-of-00003.safetensors:  77% 830M/1.08G [00:22<00:05, 46.7MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  46% 918M/1.98G [00:22<00:28, 37.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  47% 925M/1.98G [00:22<00:23, 44.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  32% 625M/1.95G [00:22<00:39, 33.2MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  33% 636M/1.95G [00:22<00:26, 49.3MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  47% 930M/1.98G [00:22<00:25, 40.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  78% 844M/1.08G [00:22<00:06, 34.3MB/s]\n",
            "model-00001-of-00003.safetensors:  33% 642M/1.95G [00:22<00:35, 36.5MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  48% 946M/1.98G [00:22<00:24, 42.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  78% 849M/1.08G [00:22<00:06, 34.1MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  48% 951M/1.98G [00:22<00:23, 44.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  34% 655M/1.95G [00:22<00:30, 42.6MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  79% 860M/1.08G [00:23<00:05, 37.9MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  48% 961M/1.98G [00:23<00:32, 31.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  80% 865M/1.08G [00:23<00:06, 32.3MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  80% 870M/1.08G [00:23<00:05, 37.7MB/s]\n",
            "model-00001-of-00003.safetensors:  34% 668M/1.95G [00:23<00:36, 34.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  81% 880M/1.08G [00:23<00:05, 38.9MB/s]\n",
            "model-00001-of-00003.safetensors:  35% 672M/1.95G [00:23<00:44, 28.7MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  49% 979M/1.98G [00:23<00:30, 33.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  82% 884M/1.08G [00:23<00:06, 31.9MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  82% 889M/1.08G [00:23<00:05, 36.0MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  50% 992M/1.98G [00:23<00:24, 41.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  83% 894M/1.08G [00:23<00:05, 34.5MB/s]\n",
            "model-00001-of-00003.safetensors:  36% 696M/1.95G [00:24<00:31, 40.1MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  50% 997M/1.98G [00:24<00:28, 34.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  83% 897M/1.08G [00:24<00:05, 31.3MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  83% 903M/1.08G [00:24<00:04, 36.8MB/s]\n",
            "model-00003-of-00003.safetensors:  84% 908M/1.08G [00:24<00:04, 36.6MB/s]\n",
            "model-00001-of-00003.safetensors:  37% 713M/1.95G [00:24<00:30, 40.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  84% 912M/1.08G [00:24<00:05, 30.3MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  51% 1.02G/1.98G [00:24<00:24, 39.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  85% 920M/1.08G [00:24<00:04, 40.3MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  52% 1.02G/1.98G [00:24<00:21, 45.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  86% 925M/1.08G [00:24<00:03, 42.3MB/s]\n",
            "model-00001-of-00003.safetensors:  38% 732M/1.95G [00:24<00:28, 42.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  86% 930M/1.08G [00:25<00:04, 34.5MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  87% 936M/1.08G [00:25<00:03, 40.1MB/s]\n",
            "model-00001-of-00003.safetensors:  38% 736M/1.95G [00:25<00:43, 27.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  52% 1.04G/1.98G [00:25<00:26, 34.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  53% 1.05G/1.98G [00:25<00:20, 45.5MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  88% 950M/1.08G [00:25<00:03, 40.4MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  88% 955M/1.08G [00:25<00:02, 42.6MB/s]\n",
            "model-00001-of-00003.safetensors:  39% 752M/1.95G [00:25<00:35, 33.7MB/s]\u001b[A\n",
            "model-00003-of-00003.safetensors:  90% 974M/1.08G [00:25<00:01, 58.1MB/s]\n",
            "model-00001-of-00003.safetensors:  39% 768M/1.95G [00:25<00:31, 37.6MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  53% 1.06G/1.98G [00:25<00:40, 23.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  91% 981M/1.08G [00:26<00:02, 45.6MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  91% 986M/1.08G [00:26<00:02, 46.9MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  92% 992M/1.08G [00:26<00:01, 48.6MB/s]\n",
            "model-00001-of-00003.safetensors:  40% 788M/1.95G [00:26<00:28, 40.2MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  92% 997M/1.08G [00:26<00:02, 39.8MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  93% 1.00G/1.08G [00:26<00:01, 44.0MB/s]\n",
            "model-00001-of-00003.safetensors:  41% 800M/1.95G [00:26<00:29, 39.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  55% 1.09G/1.98G [00:26<00:23, 38.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  93% 1.01G/1.08G [00:26<00:01, 39.7MB/s]\n",
            "model-00001-of-00003.safetensors:  42% 813M/1.95G [00:26<00:24, 46.2MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  94% 1.02G/1.08G [00:26<00:01, 44.6MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  95% 1.02G/1.08G [00:27<00:01, 44.9MB/s]\n",
            "model-00001-of-00003.safetensors:  42% 818M/1.95G [00:27<00:30, 37.5MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  56% 1.10G/1.98G [00:27<00:22, 38.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  95% 1.03G/1.08G [00:27<00:01, 39.3MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  95% 1.03G/1.08G [00:27<00:01, 40.2MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  96% 1.04G/1.08G [00:27<00:01, 43.7MB/s]\n",
            "model-00001-of-00003.safetensors:  43% 832M/1.95G [00:27<00:31, 35.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  56% 1.12G/1.98G [00:27<00:20, 42.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  96% 1.04G/1.08G [00:27<00:01, 33.1MB/s]\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  97% 1.05G/1.08G [00:27<00:00, 39.9MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  57% 1.13G/1.98G [00:27<00:21, 40.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00003-of-00003.safetensors:  98% 1.06G/1.08G [00:27<00:00, 37.1MB/s]\n",
            "model-00003-of-00003.safetensors:  99% 1.07G/1.08G [00:28<00:00, 56.0MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  58% 1.14G/1.98G [00:28<00:21, 38.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  44% 854M/1.95G [00:28<00:33, 32.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  58% 1.15G/1.98G [00:28<00:21, 39.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  44% 859M/1.95G [00:28<00:29, 36.6MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  44% 864M/1.95G [00:28<00:35, 30.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  58% 1.15G/1.98G [00:28<00:27, 30.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  45% 877M/1.95G [00:28<00:22, 47.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  59% 1.16G/1.98G [00:28<00:20, 39.8MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  59% 1.17G/1.98G [00:28<00:17, 45.8MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  45% 884M/1.95G [00:28<00:25, 42.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  59% 1.17G/1.98G [00:28<00:20, 40.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  60% 1.18G/1.98G [00:29<00:15, 50.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors:  99% 1.08G/1.08G [00:29<00:00, 18.2MB/s]\n",
            "model-00003-of-00003.safetensors: 100% 1.08G/1.08G [00:29<00:00, 20.5MB/s]\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  60% 1.19G/1.98G [00:29<00:20, 38.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  60% 1.20G/1.98G [00:29<00:15, 51.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00003-of-00003.safetensors: 100% 1.08G/1.08G [00:29<00:00, 36.7MB/s]\n",
            "\n",
            "model-00001-of-00003.safetensors:  47% 923M/1.95G [00:29<00:19, 53.5MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  61% 1.21G/1.98G [00:29<00:16, 45.8MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  48% 929M/1.95G [00:29<00:22, 44.3MB/s]\u001b[A\n",
            "\n",
            "Upload 3 LFS files:  33% 1/3 [00:29<00:59, 29.80s/it]\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  61% 1.22G/1.98G [00:29<00:16, 46.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  48% 944M/1.95G [00:29<00:16, 62.5MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  62% 1.23G/1.98G [00:29<00:13, 55.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  49% 952M/1.95G [00:30<00:19, 52.3MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  62% 1.23G/1.98G [00:30<00:15, 47.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  63% 1.24G/1.98G [00:30<00:12, 58.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  49% 960M/1.95G [00:30<00:20, 48.9MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  50% 972M/1.95G [00:30<00:15, 61.3MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  50% 979M/1.95G [00:30<00:18, 51.1MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  63% 1.25G/1.98G [00:30<00:18, 38.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  51% 987M/1.95G [00:30<00:17, 56.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  63% 1.26G/1.98G [00:30<00:16, 43.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  51% 994M/1.95G [00:30<00:19, 47.7MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  64% 1.26G/1.98G [00:30<00:20, 35.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  52% 1.01G/1.95G [00:31<00:14, 63.5MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  64% 1.27G/1.98G [00:31<00:15, 45.8MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  52% 1.01G/1.95G [00:31<00:16, 55.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  65% 1.28G/1.98G [00:31<00:17, 40.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  65% 1.29G/1.98G [00:31<00:11, 58.8MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  53% 1.02G/1.95G [00:31<00:20, 45.3MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  53% 1.04G/1.95G [00:31<00:14, 62.1MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  66% 1.30G/1.98G [00:31<00:14, 48.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  54% 1.05G/1.95G [00:31<00:18, 49.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  66% 1.31G/1.98G [00:31<00:14, 45.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  67% 1.32G/1.98G [00:32<00:12, 51.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  54% 1.06G/1.95G [00:32<00:18, 48.2MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  55% 1.07G/1.95G [00:32<00:15, 58.6MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  67% 1.33G/1.98G [00:32<00:14, 44.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  68% 1.34G/1.98G [00:32<00:10, 63.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  55% 1.07G/1.95G [00:32<00:20, 43.5MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  56% 1.09G/1.95G [00:32<00:15, 57.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  68% 1.35G/1.98G [00:32<00:13, 47.8MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  56% 1.10G/1.95G [00:32<00:16, 50.2MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  69% 1.36G/1.98G [00:32<00:14, 42.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  69% 1.37G/1.98G [00:33<00:10, 58.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  57% 1.10G/1.95G [00:33<00:18, 44.9MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  57% 1.12G/1.95G [00:33<00:13, 61.1MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  70% 1.38G/1.98G [00:33<00:11, 52.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  70% 1.39G/1.98G [00:33<00:12, 45.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  58% 1.13G/1.95G [00:33<00:19, 42.2MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  71% 1.40G/1.98G [00:33<00:11, 52.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  58% 1.13G/1.95G [00:33<00:17, 46.5MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  71% 1.41G/1.98G [00:33<00:12, 45.8MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  72% 1.42G/1.98G [00:33<00:08, 64.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  59% 1.14G/1.95G [00:33<00:22, 35.8MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  72% 1.43G/1.98G [00:34<00:11, 49.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  59% 1.15G/1.95G [00:34<00:22, 34.8MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  73% 1.44G/1.98G [00:34<00:11, 46.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  60% 1.16G/1.95G [00:34<00:17, 45.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  73% 1.45G/1.98G [00:34<00:10, 50.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  60% 1.17G/1.95G [00:34<00:20, 37.3MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  73% 1.46G/1.98G [00:34<00:10, 48.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  61% 1.18G/1.95G [00:34<00:16, 45.8MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  74% 1.47G/1.98G [00:34<00:09, 56.5MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  74% 1.47G/1.98G [00:34<00:08, 56.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  61% 1.19G/1.95G [00:35<00:18, 41.4MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  62% 1.20G/1.95G [00:35<00:12, 57.8MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  75% 1.48G/1.98G [00:35<00:11, 45.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  62% 1.21G/1.95G [00:35<00:14, 50.6MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  75% 1.49G/1.98G [00:35<00:13, 36.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  62% 1.22G/1.95G [00:35<00:16, 44.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  76% 1.50G/1.98G [00:35<00:09, 50.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  63% 1.22G/1.95G [00:35<00:14, 50.6MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  76% 1.51G/1.98G [00:35<00:10, 44.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  63% 1.23G/1.95G [00:36<00:18, 39.3MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  64% 1.25G/1.95G [00:36<00:12, 57.1MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  77% 1.52G/1.98G [00:36<00:10, 42.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  77% 1.53G/1.98G [00:36<00:07, 58.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  64% 1.26G/1.95G [00:36<00:13, 49.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  78% 1.54G/1.98G [00:36<00:08, 50.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  65% 1.26G/1.95G [00:36<00:14, 46.3MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  65% 1.27G/1.95G [00:36<00:12, 52.7MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  78% 1.55G/1.98G [00:36<00:09, 43.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  66% 1.28G/1.95G [00:36<00:14, 47.3MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  79% 1.57G/1.98G [00:36<00:07, 57.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  66% 1.29G/1.95G [00:36<00:12, 51.5MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  79% 1.57G/1.98G [00:37<00:07, 53.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  80% 1.58G/1.98G [00:37<00:07, 50.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  67% 1.30G/1.95G [00:37<00:19, 33.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  81% 1.60G/1.98G [00:37<00:05, 67.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  67% 1.31G/1.95G [00:37<00:14, 45.2MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  81% 1.61G/1.98G [00:37<00:06, 58.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  67% 1.31G/1.95G [00:37<00:14, 42.9MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  68% 1.32G/1.95G [00:37<00:11, 53.5MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  82% 1.62G/1.98G [00:37<00:07, 48.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  82% 1.63G/1.98G [00:37<00:05, 64.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  68% 1.33G/1.95G [00:38<00:13, 46.3MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  69% 1.34G/1.95G [00:38<00:10, 57.9MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  69% 1.35G/1.95G [00:38<00:12, 48.0MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  70% 1.36G/1.95G [00:38<00:12, 47.0MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  71% 1.37G/1.95G [00:38<00:08, 64.4MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  71% 1.38G/1.95G [00:38<00:10, 52.8MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  71% 1.39G/1.95G [00:39<00:12, 44.4MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  72% 1.41G/1.95G [00:39<00:08, 60.8MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  73% 1.42G/1.95G [00:39<00:09, 55.7MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  73% 1.42G/1.95G [00:39<00:11, 46.0MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  74% 1.44G/1.95G [00:39<00:08, 61.9MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  74% 1.45G/1.95G [00:40<00:08, 57.4MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  75% 1.46G/1.95G [00:40<00:10, 47.1MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  76% 1.47G/1.95G [00:40<00:07, 63.7MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  76% 1.48G/1.95G [00:40<00:08, 57.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  83% 1.64G/1.98G [00:40<00:32, 10.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  76% 1.49G/1.95G [00:40<00:09, 46.2MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  83% 1.65G/1.98G [00:41<00:24, 13.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  77% 1.50G/1.95G [00:41<00:07, 59.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  84% 1.66G/1.98G [00:41<00:18, 17.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  77% 1.51G/1.95G [00:41<00:08, 52.1MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  84% 1.66G/1.98G [00:41<00:16, 19.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  84% 1.67G/1.98G [00:41<00:12, 25.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  78% 1.52G/1.95G [00:41<00:08, 47.9MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  79% 1.54G/1.95G [00:41<00:06, 64.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  85% 1.68G/1.98G [00:41<00:11, 26.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  86% 1.69G/1.98G [00:41<00:07, 40.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  79% 1.54G/1.95G [00:41<00:07, 51.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  86% 1.70G/1.98G [00:42<00:07, 38.7MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  80% 1.55G/1.95G [00:42<00:08, 46.0MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  80% 1.57G/1.95G [00:42<00:06, 62.2MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  86% 1.71G/1.98G [00:42<00:07, 33.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  87% 1.72G/1.98G [00:42<00:05, 44.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  81% 1.57G/1.95G [00:42<00:07, 51.1MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  81% 1.58G/1.95G [00:42<00:07, 52.2MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  87% 1.73G/1.98G [00:42<00:06, 37.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  82% 1.59G/1.95G [00:42<00:07, 45.5MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  88% 1.74G/1.98G [00:42<00:05, 44.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  82% 1.59G/1.95G [00:42<00:07, 47.6MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  82% 1.60G/1.95G [00:43<00:08, 39.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  88% 1.75G/1.98G [00:43<00:06, 37.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  83% 1.61G/1.95G [00:43<00:07, 47.7MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  89% 1.75G/1.98G [00:43<00:05, 44.5MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  83% 1.62G/1.95G [00:43<00:06, 52.0MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  89% 1.76G/1.98G [00:43<00:05, 37.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  83% 1.62G/1.95G [00:43<00:08, 39.2MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  89% 1.77G/1.98G [00:43<00:04, 50.8MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  84% 1.63G/1.95G [00:43<00:07, 42.6MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  90% 1.78G/1.98G [00:43<00:04, 41.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  84% 1.63G/1.95G [00:43<00:08, 36.6MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  90% 1.79G/1.98G [00:44<00:03, 53.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  84% 1.64G/1.95G [00:44<00:07, 42.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  91% 1.80G/1.98G [00:44<00:03, 46.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  85% 1.65G/1.95G [00:44<00:07, 41.5MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  85% 1.66G/1.95G [00:44<00:05, 48.8MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  91% 1.81G/1.98G [00:44<00:03, 45.8MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  92% 1.82G/1.98G [00:44<00:02, 59.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  85% 1.66G/1.95G [00:44<00:06, 43.3MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  86% 1.68G/1.95G [00:44<00:04, 63.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  92% 1.83G/1.98G [00:44<00:03, 48.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  93% 1.84G/1.98G [00:44<00:02, 58.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  87% 1.69G/1.95G [00:44<00:05, 51.8MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  87% 1.69G/1.95G [00:45<00:04, 58.3MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  93% 1.85G/1.98G [00:45<00:02, 56.3MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  94% 1.85G/1.98G [00:45<00:02, 62.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  87% 1.70G/1.95G [00:45<00:04, 49.4MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  94% 1.86G/1.98G [00:45<00:02, 49.9MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  88% 1.71G/1.95G [00:45<00:04, 48.1MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  88% 1.72G/1.95G [00:45<00:03, 60.9MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  94% 1.87G/1.98G [00:45<00:02, 46.6MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  95% 1.89G/1.98G [00:45<00:01, 65.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  89% 1.73G/1.95G [00:45<00:04, 48.5MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  96% 1.90G/1.98G [00:45<00:01, 60.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  89% 1.74G/1.95G [00:45<00:03, 55.0MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  96% 1.90G/1.98G [00:46<00:01, 58.5MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  90% 1.75G/1.95G [00:46<00:04, 46.4MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  90% 1.76G/1.95G [00:46<00:03, 60.8MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  96% 1.91G/1.98G [00:46<00:01, 45.5MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  97% 1.92G/1.98G [00:46<00:01, 56.5MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  91% 1.77G/1.95G [00:46<00:03, 52.5MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  97% 1.93G/1.98G [00:46<00:01, 47.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  91% 1.78G/1.95G [00:46<00:03, 45.0MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  92% 1.79G/1.95G [00:46<00:02, 62.0MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  98% 1.94G/1.98G [00:46<00:01, 42.0MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  98% 1.95G/1.98G [00:46<00:00, 59.4MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  92% 1.80G/1.95G [00:47<00:03, 45.1MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  99% 1.96G/1.98G [00:47<00:00, 45.1MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00001-of-00003.safetensors:  93% 1.81G/1.95G [00:47<00:03, 40.2MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  94% 1.82G/1.95G [00:47<00:02, 56.7MB/s]\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors:  99% 1.97G/1.98G [00:47<00:00, 37.2MB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "model-00002-of-00003.safetensors: 100% 1.98G/1.98G [00:47<00:00, 51.5MB/s]\u001b[A\u001b[A\u001b[A\n",
            "model-00002-of-00003.safetensors: 100% 1.98G/1.98G [00:47<00:00, 41.4MB/s]\n",
            "\n",
            "model-00001-of-00003.safetensors:  94% 1.84G/1.95G [00:48<00:02, 43.6MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  95% 1.85G/1.95G [00:48<00:01, 59.3MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  96% 1.86G/1.95G [00:48<00:01, 46.9MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  96% 1.87G/1.95G [00:48<00:01, 45.5MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  97% 1.89G/1.95G [00:48<00:01, 61.4MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  97% 1.89G/1.95G [00:49<00:01, 53.1MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  98% 1.90G/1.95G [00:49<00:00, 47.8MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  98% 1.92G/1.95G [00:49<00:00, 63.5MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors:  99% 1.93G/1.95G [00:49<00:00, 55.4MB/s]\u001b[A\n",
            "model-00001-of-00003.safetensors: 100% 1.95G/1.95G [00:50<00:00, 38.9MB/s]\n",
            "\n",
            "\n",
            "Upload 3 LFS files: 100% 3/3 [00:50<00:00, 16.80s/it]\n",
            "[INFO|tokenization_utils_base.py:2513] 2024-06-02 02:07:54,530 >> tokenizer config file saved in gemma_lora_merged/tokenizer_config.json\n",
            "[INFO|tokenization_utils_base.py:2522] 2024-06-02 02:07:54,530 >> Special tokens file saved in gemma_lora_merged/special_tokens_map.json\n",
            "README.md: 100% 5.19k/5.19k [00:00<00:00, 16.0MB/s]\n",
            "[INFO|tokenization_utils_base.py:2513] 2024-06-02 02:07:55,448 >> tokenizer config file saved in /tmp/tmphfp90355/tokenizer_config.json\n",
            "[INFO|tokenization_utils_base.py:2522] 2024-06-02 02:07:55,448 >> Special tokens file saved in /tmp/tmphfp90355/special_tokens_map.json\n",
            "[INFO|hub.py:759] 2024-06-02 02:07:56,019 >> Uploading the following files to windmaple/gemma-2b-finetuned-model-llama-factory: special_tokens_map.json,tokenizer.model,README.md,tokenizer_config.json,tokenizer.json\n",
            "tokenizer.model:   0% 0.00/4.24M [00:00<?, ?B/s]\n",
            "tokenizer.json:   0% 0.00/17.5M [00:00<?, ?B/s]\u001b[A\n",
            "\n",
            "Upload 2 LFS files:   0% 0/2 [00:00<?, ?it/s]\u001b[A\u001b[A\n",
            "tokenizer.model: 100% 4.24M/4.24M [00:00<00:00, 23.6MB/s]\n",
            "\n",
            "\n",
            "tokenizer.json: 100% 17.5M/17.5M [00:00<00:00, 40.0MB/s]\n",
            "\n",
            "\n",
            "Upload 2 LFS files: 100% 2/2 [00:00<00:00,  3.25it/s]\n"
          ]
        }
      ],
      "source": [
        "import json\n",
        "\n",
        "args = dict(\n",
        "    model_name_or_path=\"google/gemma-2b\",  # use official non-quantized Gemma 2B model\n",
        "    adapter_name_or_path=\"gemma_lora\",  # load the saved LoRA adapters\n",
        "    template=\"gemma\",  # same to the one in training\n",
        "    finetuning_type=\"lora\",  # same to the one in training\n",
        "    export_dir=\"gemma_lora_merged\",  # path to save the merged model\n",
        "    export_size=2,  # the file shard size (in GB) of the merged model\n",
        "    export_device=\"cpu\",  # the device used in export, can be chosen from `cpu` and `cuda`\n",
        "    export_hub_model_id=\"gemma-2b-finetuned-model-llama-factory\",  # your Hugging Face hub model ID\n",
        ")\n",
        "\n",
        "json.dump(args, open(\"merge_gemma.json\", \"w\", encoding=\"utf-8\"), indent=2)\n",
        "\n",
        "%cd /content/LLaMA-Factory/\n",
        "\n",
        "!llamafactory-cli export merge_gemma.json"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "kM8NDYemuceH"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0da44d3e21044e4c9ea38a0ec42d5a0d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b9df7df513fe4810addfbefae98dc4b9",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3da14b51edc64eb3b1e5a1d47e438fcc",
            "value": 2
          }
        },
        "14e07cff44704835a0a7bf2bd09df670": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "35b7447f99714f8093d0def80702b8ce": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_53d852152fe742baa7dd81f5af020796",
            "placeholder": "​",
            "style": "IPY_MODEL_f984f9e1d31e4c07813bf44ef5c61925",
            "value": "Loading checkpoint shards: 100%"
          }
        },
        "39404874553f41aab27788d96e122f35": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bdd56d50debc4060b26cfcf17b7c638a",
            "placeholder": "​",
            "style": "IPY_MODEL_14e07cff44704835a0a7bf2bd09df670",
            "value": " 2/2 [00:06&lt;00:00,  2.55s/it]"
          }
        },
        "3d1b6ab8ca8e4bb5978fe2b32d9f09bc": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_35b7447f99714f8093d0def80702b8ce",
              "IPY_MODEL_0da44d3e21044e4c9ea38a0ec42d5a0d",
              "IPY_MODEL_39404874553f41aab27788d96e122f35"
            ],
            "layout": "IPY_MODEL_8cf8781120ab40b09de84b78b9e38726"
          }
        },
        "3da14b51edc64eb3b1e5a1d47e438fcc": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "53d852152fe742baa7dd81f5af020796": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8cf8781120ab40b09de84b78b9e38726": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b9df7df513fe4810addfbefae98dc4b9": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bdd56d50debc4060b26cfcf17b7c638a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f984f9e1d31e4c07813bf44ef5c61925": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}