examples/ChatGLM2_LoRA注释版.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0fuBIdB8XDVD"
   },
   "source": [
    "# ChatGLM2-6b微调保姆级教程 - Colab 版本\n",
    "\n",
    "- 本notebook适配了Colab的一些tricy issue\n",
    "- 本notebook对代码进行了较为详细的解释\n",
    "- Thanks for [有毅力的吃货](https://github.com/lyhue1991/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lZfFNQfQ2Ld-"
   },
   "source": [
    "## 安装环境\n",
    "\n",
    "- 注意:与ChatGLM2 相关的依赖使用官方提供的requirement.txt进行安装\n",
    "- 使用命令行执行`pip install -r requirements.txt`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CoNKd5M08lOh"
   },
   "outputs": [],
   "source": [
    "#安装环境\n",
    "\n",
    "#chatglm install by terminal\n",
    "# !pip install -r requirements.txt\n",
    "\n",
    "#finetune\n",
    "!pip install -U accelerate\n",
    "!pip install datasets\n",
    "!pip install -U peft\n",
    "!pip install -U torchkeras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Lfp6upkf2cOb"
   },
   "source": [
    "## 展示模型没有进行Finetune之前的能力"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LD-h2a_C8uMG"
   },
   "outputs": [],
   "source": [
    "from transformers import  AutoModel,AutoTokenizer\n",
    "model_name = 'THUDM/chatglm2-6b'\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)\n",
    "# 这里将模型转化成了半精度FP16模式，可以显存优化，可以减少模型在GPU上的显存使用量，但可能会牺牲一定的精度\n",
    "# 如果你希望使用更高的精度，同时你有足够好的GPU你可以移除这个函数的调用\n",
    "model = AutoModel.from_pretrained(model_name,trust_remote_code=True).half().cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "u4Zks4Gg-B3M"
   },
   "outputs": [],
   "source": [
    "# define a basic prompt for express comment\n",
    "# prompt template with 4shots ahead\n",
    "\n",
    "prompt = \"\"\"文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者差评。\n",
    "\n",
    "下面是一些范例:\n",
    "\n",
    "味道真不错 -> 好评\n",
    "太辣了，吃不下都  -> 差评\n",
    "吃完拉肚子了 -> 差评\n",
    "味道好吃 -> 好评\n",
    "\n",
    "请对下述评论进行分类。返回'好评'或者'差评'，无需其它说明和解释。\n",
    "\n",
    "xxxxxx ->\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "# replace query with real content\n",
    "\n",
    "def get_prompt(text):\n",
    "    return prompt.replace('xxxxxx',text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8NGotBvPo_Wg",
    "outputId": "40efc686-d736-4812-ffac-3341e65faa47"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "好评\n",
      "[(\"文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者差评。\\n\\n下面是一些范例:\\n\\n味道真不错 -> 好评\\n太辣了，吃不下都  -> 差评\\n吃完拉肚子了 -> 差评\\n味道好吃 -> 好评\\n\\n请对下述评论进行分类。返回'好评'或者'差评'，无需其它说明和解释。\\n\\n味道不错，下次还来 ->\\n\\n\", '好评')]\n"
     ]
    }
   ],
   "source": [
    "# try for once and test model workable\n",
    "response, his = model.chat(tokenizer, get_prompt('味道不错，下次还来'), history=[])\n",
    "print(response)\n",
    "print(his)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fXngFyCftLZ2",
    "outputId": "e4b4eaf9-b5aa-4aba-cf8e-267410f18479"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(\"文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者差评。\\n\\n下面是一些范例:\\n\\n味道真不错 -> 好评\\n太辣了，吃不下都  -> 差评\\n吃完拉肚子了 -> 差评\\n味道好吃 -> 好评\\n\\n请对下述评论进行分类。返回'好评'或者'差评'，无需其它说明和解释。\\n\\n味道不错，下次还来 ->\\n\\n\", '好评'), ('太贵了 -> ', '差评'), ('非常快，味道好 -> ', '好评'), ('这么咸真的是醉了 -> ', '差评'), ('价格感人 优惠多多 -> ', '好评')]\n"
     ]
    }
   ],
   "source": [
    "# add new history comment as new shots\n",
    "his.append((\"太贵了 -> \",\"差评\"))\n",
    "his.append((\"非常快，味道好 -> \",\"好评\"))\n",
    "\n",
    "his.append((\"这么咸真的是醉了 -> \",\"差评\"))\n",
    "his.append((\"价格感人 优惠多多 -> \",\"好评\"))\n",
    "# now we got 8 shots\n",
    "print(his)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pRIRtWYYtiE2",
    "outputId": "4bc7b8ad-477c-46d2-e0e7-b103e80a37b7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "差评\n",
      "差评\n",
      "好评\n"
     ]
    }
   ],
   "source": [
    "# based on new shots to make prediction\n",
    "response, his = model.chat(tokenizer, \"一言难尽啊 -> \", history=his)\n",
    "print(response)\n",
    "\n",
    "response, his = model.chat(tokenizer, \"还凑合一般般 -> \", history=his)\n",
    "print(response)  # 预测有失偏颇\n",
    "\n",
    "response, his = model.chat(tokenizer, \"我家狗狗爱吃的 -> \", history=his)\n",
    "print(response)  # 预测有失偏颇"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "mSu3-OkRufc7",
    "outputId": "39fea102-22b5-46e1-e2e8-fb7f09e91ca8"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'差评'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# define a function for above\n",
    "# only return prediction result rather than result & history\n",
    "def predict(text):\n",
    "    response, history = model.chat(tokenizer, f\"{text} ->\", history=his,\n",
    "    temperature=0.01)\n",
    "    return response\n",
    "\n",
    "predict('死鬼，咋弄得这么有滋味呢') # try for once and got a incompatiable result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "DP5Oc8lsutH0",
    "outputId": "63c4b4a2-ebfe-4d08-ed9e-5f3157e10150"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\n",
       "  <div id=\"df-2ca2fbd0-db9f-4710-b438-c46c40f778c4\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>很快，好吃，味道足，量大</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>没有送水没有送水没有送水</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>非常快，态度好。</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>方便，快捷，味道可口，快递给力</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>菜味道很棒！送餐很及时！</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2ca2fbd0-db9f-4710-b438-c46c40f778c4')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "\n",
       "\n",
       "\n",
       "    <div id=\"df-45474fce-7692-437d-b899-f074bee0a8df\">\n",
       "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-45474fce-7692-437d-b899-f074bee0a8df')\"\n",
       "              title=\"Suggest charts.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "      </button>\n",
       "    </div>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "    background-color: #E8F0FE;\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: #1967D2;\n",
       "    height: 32px;\n",
       "    padding: 0 0 0 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: #E2EBFA;\n",
       "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: #174EA6;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "    background-color: #3B4455;\n",
       "    fill: #D2E3FC;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart:hover {\n",
       "    background-color: #434B5C;\n",
       "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "    fill: #FFFFFF;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "    <script>\n",
       "      async function quickchart(key) {\n",
       "        const containerElement = document.querySelector('#' + key);\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      }\n",
       "    </script>\n",
       "\n",
       "      <script>\n",
       "\n",
       "function displayQuickchartButton(domScope) {\n",
       "  let quickchartButtonEl =\n",
       "    domScope.querySelector('#df-45474fce-7692-437d-b899-f074bee0a8df button.colab-df-quickchart');\n",
       "  quickchartButtonEl.style.display =\n",
       "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "}\n",
       "\n",
       "        displayQuickchartButton(document);\n",
       "      </script>\n",
       "      <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-2ca2fbd0-db9f-4710-b438-c46c40f778c4 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-2ca2fbd0-db9f-4710-b438-c46c40f778c4');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "   label           review\n",
       "0      1     很快，好吃，味道足，量大\n",
       "1      1     没有送水没有送水没有送水\n",
       "2      1         非常快，态度好。\n",
       "3      1  方便，快捷，味道可口，快递给力\n",
       "4      1     菜味道很棒！送餐很及时！"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import datasets\n",
    "\n",
    "df = pd.read_csv(\"waimai_10k.csv\")\n",
    "\n",
    "df.head() # 0 是差评，1是好评"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "TKStP1egwQz9",
    "outputId": "f4d14945-218c-4282-d0f4-de4db8cca8a2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "好评    4000\n",
      "差评    4000\n",
      "Name: tag, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "df['tag'] = df['label'].map({0:'差评',1:'好评'}) # create a new column for label\n",
    "df = df.rename({'review':'text'},axis = 1)\n",
    "\n",
    "dfgood = df.query('tag==\"好评\"')\n",
    "dfbad = df.query('tag==\"差评\"').head(len(dfgood)) #采样部分差评，让好评差评平衡\n",
    "df = pd.concat([dfgood,dfbad]) # 重新制作数据集确保数据平衡\n",
    "print(df['tag'].value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "2QSds6vMw2KO"
   },
   "outputs": [],
   "source": [
    "# 将Pandas DataFrame对象转换为Hugging Face Dataset对象\n",
    "# 将数据集划分为训练集和测试集\n",
    "# test_size: 指定测试集的大小。在这个例子中，测试集的大小是2000，这意味着2000个样本将被用于测试集\n",
    "# 已知完整的数据集一共包括8000个案例，因此有6000个训练集\n",
    "# shuffle: 如果设置为True，数据集在划分之前会被打乱，可以确保训练集和测试集是随机抽样的，从而避免因样本排序引起的偏差\n",
    "# seed: 设置随机数生成器的种子，用于控制随机打乱的过程，确保结果是可复制的。在这里，种子被设置为43，你也可以设置为其他的数字\n",
    "ds_dic = datasets.Dataset.from_pandas(df).train_test_split(\n",
    "    test_size = 2000,shuffle=True, seed = 43)\n",
    "dftrain = ds_dic['train'].to_pandas() # 将训练集转化为pandas\n",
    "dftest = ds_dic['test'].to_pandas() # 将验证集转化为pandas\n",
    "# 用于将DataFrame对象保存为Parquet格式的文件\n",
    "dftrain.to_parquet('dftrain.parquet')\n",
    "dftest.to_parquet('dftest.parquet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0PqMB42Bw2EL",
    "outputId": "cd54fef3-67b6-4df0-d855-a32cd638651c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000\n"
     ]
    }
   ],
   "source": [
    "preds = ['' for x in dftest['tag']] # 基于测试集的长度创建一个新的数组，用于存储测试的结果\n",
    "print(len(preds))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "SAzFEOaHzGeW",
    "outputId": "e4b08f02-9792-4ec7-b33c-bfa45ae0977b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\n",
       "  <div id=\"df-aca0fa20-ccfb-4312-ab80-c14af6d20e34\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "      <th>tag</th>\n",
       "      <th>__index_level_0__</th>\n",
       "      <th>pred</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>我的两个中奖瓶盖拿走后就没有下文了，4元可乐居然是灌装的！！！！</td>\n",
       "      <td>差评</td>\n",
       "      <td>6667</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>很好喜欢极力推荐噢</td>\n",
       "      <td>好评</td>\n",
       "      <td>2269</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>商家不把顾客留言放心上，强调不要辣椒，商家还是放了很了很多，弄的孩子无法吃，其他菜品味道不错...</td>\n",
       "      <td>差评</td>\n",
       "      <td>7169</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>速度很快，食品也特别棒！</td>\n",
       "      <td>好评</td>\n",
       "      <td>3687</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>送餐时间，1个小时20分钟！！！！！！电话催单，直接挂客户电话！！！！什么破服务态度！！~！...</td>\n",
       "      <td>差评</td>\n",
       "      <td>6320</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-aca0fa20-ccfb-4312-ab80-c14af6d20e34')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "\n",
       "\n",
       "\n",
       "    <div id=\"df-27897958-2dca-49e6-a7d7-bde8aca173b7\">\n",
       "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-27897958-2dca-49e6-a7d7-bde8aca173b7')\"\n",
       "              title=\"Suggest charts.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "      </button>\n",
       "    </div>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "    background-color: #E8F0FE;\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: #1967D2;\n",
       "    height: 32px;\n",
       "    padding: 0 0 0 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: #E2EBFA;\n",
       "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: #174EA6;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "    background-color: #3B4455;\n",
       "    fill: #D2E3FC;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart:hover {\n",
       "    background-color: #434B5C;\n",
       "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "    fill: #FFFFFF;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "    <script>\n",
       "      async function quickchart(key) {\n",
       "        const containerElement = document.querySelector('#' + key);\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      }\n",
       "    </script>\n",
       "\n",
       "      <script>\n",
       "\n",
       "function displayQuickchartButton(domScope) {\n",
       "  let quickchartButtonEl =\n",
       "    domScope.querySelector('#df-27897958-2dca-49e6-a7d7-bde8aca173b7 button.colab-df-quickchart');\n",
       "  quickchartButtonEl.style.display =\n",
       "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "}\n",
       "\n",
       "        displayQuickchartButton(document);\n",
       "      </script>\n",
       "      <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-aca0fa20-ccfb-4312-ab80-c14af6d20e34 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-aca0fa20-ccfb-4312-ab80-c14af6d20e34');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "   label                                               text tag  \\\n",
       "0      0                   我的两个中奖瓶盖拿走后就没有下文了，4元可乐居然是灌装的！！！！  差评   \n",
       "1      1                                          很好喜欢极力推荐噢  好评   \n",
       "2      0  商家不把顾客留言放心上，强调不要辣椒，商家还是放了很了很多，弄的孩子无法吃，其他菜品味道不错...  差评   \n",
       "3      1                                       速度很快，食品也特别棒！  好评   \n",
       "4      0  送餐时间，1个小时20分钟！！！！！！电话催单，直接挂客户电话！！！！什么破服务态度！！~！...  差评   \n",
       "\n",
       "   __index_level_0__ pred  \n",
       "0               6667       \n",
       "1               2269       \n",
       "2               7169       \n",
       "3               3687       \n",
       "4               6320       "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dftest['pred'] = preds\n",
    "dftest.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "sU3RWN6szyTz",
    "outputId": "1ea3d4df-45ca-465e-de1f-d8340a6ef8e9"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2000/2000 [04:14<00:00,  7.86it/s]\n"
     ]
    }
   ],
   "source": [
    "# 对dftest中的每一行文本进行预测，然后把预测结果存储在preds列表中\n",
    "# 使用当前的未经过ft的模型进行评论类型的预测\n",
    "from tqdm import tqdm\n",
    "for i in tqdm(range(len(dftest))):\n",
    "    text = dftest['text'].loc[i]\n",
    "    preds[i] = predict(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "nn8vYLHj1D5N",
    "outputId": "145c3fe3-1b26-41ae-fc2e-41085322520b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\n",
       "  <div id=\"df-5b7454a6-ef6c-488f-8da6-f537351c09e3\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "      <th>tag</th>\n",
       "      <th>__index_level_0__</th>\n",
       "      <th>pred</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>我的两个中奖瓶盖拿走后就没有下文了，4元可乐居然是灌装的！！！！</td>\n",
       "      <td>差评</td>\n",
       "      <td>6667</td>\n",
       "      <td>差评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>很好喜欢极力推荐噢</td>\n",
       "      <td>好评</td>\n",
       "      <td>2269</td>\n",
       "      <td>好评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>商家不把顾客留言放心上，强调不要辣椒，商家还是放了很了很多，弄的孩子无法吃，其他菜品味道不错...</td>\n",
       "      <td>差评</td>\n",
       "      <td>7169</td>\n",
       "      <td>差评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>速度很快，食品也特别棒！</td>\n",
       "      <td>好评</td>\n",
       "      <td>3687</td>\n",
       "      <td>好评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>送餐时间，1个小时20分钟！！！！！！电话催单，直接挂客户电话！！！！什么破服务态度！！~！...</td>\n",
       "      <td>差评</td>\n",
       "      <td>6320</td>\n",
       "      <td>差评</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5b7454a6-ef6c-488f-8da6-f537351c09e3')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "\n",
       "\n",
       "\n",
       "    <div id=\"df-0bb15137-3966-47f7-922b-e050f546dec6\">\n",
       "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-0bb15137-3966-47f7-922b-e050f546dec6')\"\n",
       "              title=\"Suggest charts.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "      </button>\n",
       "    </div>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "    background-color: #E8F0FE;\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: #1967D2;\n",
       "    height: 32px;\n",
       "    padding: 0 0 0 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: #E2EBFA;\n",
       "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: #174EA6;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "    background-color: #3B4455;\n",
       "    fill: #D2E3FC;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart:hover {\n",
       "    background-color: #434B5C;\n",
       "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "    fill: #FFFFFF;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "    <script>\n",
       "      async function quickchart(key) {\n",
       "        const containerElement = document.querySelector('#' + key);\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      }\n",
       "    </script>\n",
       "\n",
       "      <script>\n",
       "\n",
       "function displayQuickchartButton(domScope) {\n",
       "  let quickchartButtonEl =\n",
       "    domScope.querySelector('#df-0bb15137-3966-47f7-922b-e050f546dec6 button.colab-df-quickchart');\n",
       "  quickchartButtonEl.style.display =\n",
       "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "}\n",
       "\n",
       "        displayQuickchartButton(document);\n",
       "      </script>\n",
       "      <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-5b7454a6-ef6c-488f-8da6-f537351c09e3 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-5b7454a6-ef6c-488f-8da6-f537351c09e3');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "   label                                               text tag  \\\n",
       "0      0                   我的两个中奖瓶盖拿走后就没有下文了，4元可乐居然是灌装的！！！！  差评   \n",
       "1      1                                          很好喜欢极力推荐噢  好评   \n",
       "2      0  商家不把顾客留言放心上，强调不要辣椒，商家还是放了很了很多，弄的孩子无法吃，其他菜品味道不错...  差评   \n",
       "3      1                                       速度很快，食品也特别棒！  好评   \n",
       "4      0  送餐时间，1个小时20分钟！！！！！！电话催单，直接挂客户电话！！！！什么破服务态度！！~！...  差评   \n",
       "\n",
       "   __index_level_0__ pred  \n",
       "0               6667   差评  \n",
       "1               2269   好评  \n",
       "2               7169   差评  \n",
       "3               3687   好评  \n",
       "4               6320   差评  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dftest['pred'] = preds\n",
    "dftest.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 205
    },
    "id": "GrFeZ7uWzTkj",
    "outputId": "6034f6ee-c15a-4613-e713-24f02bf0eed2"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\n",
       "  <div id=\"df-7c59207f-b7d6-434a-905b-c7c4a4a4640c\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>pred</th>\n",
       "      <th>可能是评论者点的菜品中有带头发的东西,导致自己家狗狗误食了。因此,评论者给了差评。</th>\n",
       "      <th>好评</th>\n",
       "      <th>好评， 加油</th>\n",
       "      <th>差评</th>\n",
       "      <th>无法确定评价,因为只提供了送到时间,但没有提到是否准时到达。</th>\n",
       "      <th>负面</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tag</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>好评</th>\n",
       "      <td>NaN</td>\n",
       "      <td>809.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>183.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>差评</th>\n",
       "      <td>1.0</td>\n",
       "      <td>54.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>950.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7c59207f-b7d6-434a-905b-c7c4a4a4640c')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "\n",
       "\n",
       "\n",
       "    <div id=\"df-21de59cb-6167-44af-ac9b-a3963f5f1623\">\n",
       "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-21de59cb-6167-44af-ac9b-a3963f5f1623')\"\n",
       "              title=\"Suggest charts.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "      </button>\n",
       "    </div>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "    background-color: #E8F0FE;\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: #1967D2;\n",
       "    height: 32px;\n",
       "    padding: 0 0 0 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: #E2EBFA;\n",
       "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: #174EA6;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "    background-color: #3B4455;\n",
       "    fill: #D2E3FC;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart:hover {\n",
       "    background-color: #434B5C;\n",
       "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "    fill: #FFFFFF;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "    <script>\n",
       "      async function quickchart(key) {\n",
       "        const containerElement = document.querySelector('#' + key);\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      }\n",
       "    </script>\n",
       "\n",
       "      <script>\n",
       "\n",
       "function displayQuickchartButton(domScope) {\n",
       "  let quickchartButtonEl =\n",
       "    domScope.querySelector('#df-21de59cb-6167-44af-ac9b-a3963f5f1623 button.colab-df-quickchart');\n",
       "  quickchartButtonEl.style.display =\n",
       "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "}\n",
       "\n",
       "        displayQuickchartButton(document);\n",
       "      </script>\n",
       "      <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-7c59207f-b7d6-434a-905b-c7c4a4a4640c button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-7c59207f-b7d6-434a-905b-c7c4a4a4640c');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "pred  可能是评论者点的菜品中有带头发的东西,导致自己家狗狗误食了。因此,评论者给了差评。     好评  好评， 加油     差评  \\\n",
       "tag                                                                     \n",
       "好评                                          NaN  809.0     1.0  183.0   \n",
       "差评                                          1.0   54.0     NaN  950.0   \n",
       "\n",
       "pred  无法确定评价,因为只提供了送到时间,但没有提到是否准时到达。   负面  \n",
       "tag                                        \n",
       "好评                               1.0  NaN  \n",
       "差评                               NaN  1.0  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 创建 dftest 的数据透视表\n",
    "# index='tag': 将'dftest'数据集的'tag'列设置为透视表的索引（行标签）\n",
    "# columns='pred': 将数据集的'pred'列设置为透视表的列标签\n",
    "# values='text': 将数据集的'text'列的值填充到透视表中。一般来说，我们会选择一个需要进行计算的列作为values\n",
    "# aggfunc='count': 聚合函数，定义了我们如何对values进行计算。在这里，我们选择的是'count'，意味着我们要对'text'列的值进行计数\n",
    "dftest.pivot_table(index='tag',columns = 'pred',values='text',aggfunc='count')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "j5eR8Rww1xM8",
    "outputId": "55cae7e8-045a-4e3d-cf53-f4dbe17ab002"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy= 0.8795\n"
     ]
    }
   ],
   "source": [
    "acc = len(dftest.query('tag==pred'))/len(dftest) # 没有经过ft的model的accuracy为87.95，分数不高也不低\n",
    "print('accuracy=',acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bI-_2xyo2FfY"
   },
   "source": [
    "## 准备 Finetune 需要的数据\n",
    "\n",
    "\n",
    "- 需要把数据整理成对话的形式，即 context 和 target 的配对，然后拼到一起作为一条样本\n",
    "- 几乎所有的LLM都是即给定一段话的上半部分，它会去续写下半部分\n",
    "- 这里指定上半部分为设计的文本分类任务的prompt，下半部分为文本分类结果\n",
    "- 微调的目标就是让它预测的下半部分跟我们的设定的文本分类一致\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "id": "V6wO-eNX23pT"
   },
   "outputs": [],
   "source": [
    "# 加载数据\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import datasets\n",
    "\n",
    "dftrain = pd.read_parquet('dftrain.parquet')\n",
    "dftest = pd.read_parquet('dftest.parquet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "cLXegyBc2_VP",
    "outputId": "1db28b80-5198-441d-8e71-36491384a0dc"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\n",
       "  <div id=\"df-c5f55c21-5b9f-4bfb-875a-3ab235a8d42e\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "      <th>tag</th>\n",
       "      <th>__index_level_0__</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>首先送的晚,然后我要的脱脂奶不加冰,做的又加奶又加冰！！！！</td>\n",
       "      <td>差评</td>\n",
       "      <td>7810</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>送餐速度快，好吃</td>\n",
       "      <td>好评</td>\n",
       "      <td>1390</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>买了两份馅饼只给了一份，也没解释为啥也没退钱</td>\n",
       "      <td>好评</td>\n",
       "      <td>2791</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>没给我果汁!!!!!心塞</td>\n",
       "      <td>差评</td>\n",
       "      <td>6327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>很不错，服务非常好，很认真</td>\n",
       "      <td>好评</td>\n",
       "      <td>761</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c5f55c21-5b9f-4bfb-875a-3ab235a8d42e')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "\n",
       "\n",
       "\n",
       "    <div id=\"df-c9f565e2-f97f-485d-825c-af9711c7ef49\">\n",
       "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-c9f565e2-f97f-485d-825c-af9711c7ef49')\"\n",
       "              title=\"Suggest charts.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "      </button>\n",
       "    </div>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "    background-color: #E8F0FE;\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: #1967D2;\n",
       "    height: 32px;\n",
       "    padding: 0 0 0 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: #E2EBFA;\n",
       "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: #174EA6;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "    background-color: #3B4455;\n",
       "    fill: #D2E3FC;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart:hover {\n",
       "    background-color: #434B5C;\n",
       "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "    fill: #FFFFFF;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "    <script>\n",
       "      async function quickchart(key) {\n",
       "        const containerElement = document.querySelector('#' + key);\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      }\n",
       "    </script>\n",
       "\n",
       "      <script>\n",
       "\n",
       "function displayQuickchartButton(domScope) {\n",
       "  let quickchartButtonEl =\n",
       "    domScope.querySelector('#df-c9f565e2-f97f-485d-825c-af9711c7ef49 button.colab-df-quickchart');\n",
       "  quickchartButtonEl.style.display =\n",
       "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "}\n",
       "\n",
       "        displayQuickchartButton(document);\n",
       "      </script>\n",
       "      <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-c5f55c21-5b9f-4bfb-875a-3ab235a8d42e button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-c5f55c21-5b9f-4bfb-875a-3ab235a8d42e');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "   label                            text tag  __index_level_0__\n",
       "0      0  首先送的晚,然后我要的脱脂奶不加冰,做的又加奶又加冰！！！！  差评               7810\n",
       "1      1                        送餐速度快，好吃  好评               1390\n",
       "2      1          买了两份馅饼只给了一份，也没解释为啥也没退钱  好评               2791\n",
       "3      0                    没给我果汁!!!!!心塞  差评               6327\n",
       "4      1                   很不错，服务非常好，很认真  好评                761"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dftrain.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "NiarOopz2_vp",
    "outputId": "bda6fa76-2252-479d-c10e-286f959843b1"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\n",
       "  <div id=\"df-66e899f6-d711-46f1-bd5f-efa7d17b3c6c\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "      <th>tag</th>\n",
       "      <th>__index_level_0__</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>我的两个中奖瓶盖拿走后就没有下文了，4元可乐居然是灌装的！！！！</td>\n",
       "      <td>差评</td>\n",
       "      <td>6667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>很好喜欢极力推荐噢</td>\n",
       "      <td>好评</td>\n",
       "      <td>2269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>商家不把顾客留言放心上，强调不要辣椒，商家还是放了很了很多，弄的孩子无法吃，其他菜品味道不错...</td>\n",
       "      <td>差评</td>\n",
       "      <td>7169</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>速度很快，食品也特别棒！</td>\n",
       "      <td>好评</td>\n",
       "      <td>3687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>送餐时间，1个小时20分钟！！！！！！电话催单，直接挂客户电话！！！！什么破服务态度！！~！...</td>\n",
       "      <td>差评</td>\n",
       "      <td>6320</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-66e899f6-d711-46f1-bd5f-efa7d17b3c6c')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "\n",
       "\n",
       "\n",
       "    <div id=\"df-aefd72c2-c52f-4964-a199-0d9908b581b1\">\n",
       "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-aefd72c2-c52f-4964-a199-0d9908b581b1')\"\n",
       "              title=\"Suggest charts.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "      </button>\n",
       "    </div>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "    background-color: #E8F0FE;\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: #1967D2;\n",
       "    height: 32px;\n",
       "    padding: 0 0 0 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: #E2EBFA;\n",
       "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: #174EA6;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "    background-color: #3B4455;\n",
       "    fill: #D2E3FC;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart:hover {\n",
       "    background-color: #434B5C;\n",
       "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "    fill: #FFFFFF;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "    <script>\n",
       "      async function quickchart(key) {\n",
       "        const containerElement = document.querySelector('#' + key);\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      }\n",
       "    </script>\n",
       "\n",
       "      <script>\n",
       "\n",
       "function displayQuickchartButton(domScope) {\n",
       "  let quickchartButtonEl =\n",
       "    domScope.querySelector('#df-aefd72c2-c52f-4964-a199-0d9908b581b1 button.colab-df-quickchart');\n",
       "  quickchartButtonEl.style.display =\n",
       "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "}\n",
       "\n",
       "        displayQuickchartButton(document);\n",
       "      </script>\n",
       "      <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-66e899f6-d711-46f1-bd5f-efa7d17b3c6c button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-66e899f6-d711-46f1-bd5f-efa7d17b3c6c');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "   label                                               text tag  \\\n",
       "0      0                   我的两个中奖瓶盖拿走后就没有下文了，4元可乐居然是灌装的！！！！  差评   \n",
       "1      1                                          很好喜欢极力推荐噢  好评   \n",
       "2      0  商家不把顾客留言放心上，强调不要辣椒，商家还是放了很了很多，弄的孩子无法吃，其他菜品味道不错...  差评   \n",
       "3      1                                       速度很快，食品也特别棒！  好评   \n",
       "4      0  送餐时间，1个小时20分钟！！！！！！电话催单，直接挂客户电话！！！！什么破服务态度！！~！...  差评   \n",
       "\n",
       "   __index_level_0__  \n",
       "0               6667  \n",
       "1               2269  \n",
       "2               7169  \n",
       "3               3687  \n",
       "4               6320  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dftest.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "N-MmL4UR3ses",
    "outputId": "d95b9ffd-0ad8-4aa0-dcb1-6e61b55eb6bf"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "好评    3006\n",
       "差评    2994\n",
       "Name: tag, dtype: int64"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dftrain['tag'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "id": "PSt6cTED4WS8"
   },
   "outputs": [],
   "source": [
    "# 参考chatGLM2 中的输入源码\n",
    "model.build_inputs??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ajv8gvVr5VWt"
   },
   "source": [
    "```python\n",
    "def build_inputs(self, tokenizer, query: str, history: List[Tuple[str, str]] = None):\n",
    "        prompt = tokenizer.build_prompt(query, history=history)\n",
    "        inputs = tokenizer([prompt], return_tensors=\"pt\")\n",
    "        inputs = inputs.to(self.device)\n",
    "        return inputs\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "id": "Gs_Uda0s3vuH"
   },
   "outputs": [],
   "source": [
    "# 将上下文整理成与推理时候一致，参照model.chat中的源码\n",
    "def build_inputs(query, history):\n",
    "    prompt = \"\"\n",
    "    for i, (old_query, response) in enumerate(history):\n",
    "        prompt += \"[Round {}]\\n\\n问：{}\\n\\n答：{}\\n\\n\".format(i + 1, old_query, response) # history中的第几轮次，问了什么，得到了什么答案\n",
    "    prompt += \"[Round {}]\\n\\n问：{} -> \\n\\n答：\".format(len(history) + 1, query) # 当前轮次，当前问话\n",
    "    return prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "evWVCKGI6Zh6",
    "outputId": "6e135426-7200-4f81-e899-6b7205553e0a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Round 1]\n",
      "\n",
      "问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者差评。\n",
      "\n",
      "下面是一些范例:\n",
      "\n",
      "味道真不错 -> 好评\n",
      "太辣了，吃不下都  -> 差评\n",
      "吃完拉肚子了 -> 差评\n",
      "味道好吃 -> 好评\n",
      "\n",
      "请对下述评论进行分类。返回'好评'或者'差评'，无需其它说明和解释。\n",
      "\n",
      "味道不错，下次还来 ->\n",
      "\n",
      "\n",
      "\n",
      "答：好评\n",
      "\n",
      "[Round 2]\n",
      "\n",
      "问：太贵了 -> \n",
      "\n",
      "答：差评\n",
      "\n",
      "[Round 3]\n",
      "\n",
      "问：非常快，味道好 -> \n",
      "\n",
      "答：好评\n",
      "\n",
      "[Round 4]\n",
      "\n",
      "问：这么咸真的是醉了 -> \n",
      "\n",
      "答：差评\n",
      "\n",
      "[Round 5]\n",
      "\n",
      "问：价格感人 优惠多多 -> \n",
      "\n",
      "答：好评\n",
      "\n",
      "[Round 6]\n",
      "\n",
      "问：一言难尽啊 -> \n",
      "\n",
      "答：差评\n",
      "\n",
      "[Round 7]\n",
      "\n",
      "问：还凑合一般般 -> \n",
      "\n",
      "答：差评\n",
      "\n",
      "[Round 8]\n",
      "\n",
      "问：我家狗狗爱吃的 -> \n",
      "\n",
      "答：好评\n",
      "\n",
      "[Round 9]\n",
      "\n",
      "问：味道不太行 -> \n",
      "\n",
      "答：\n"
     ]
    }
   ],
   "source": [
    "print(build_inputs('味道不太行',history=his))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "mD-i3vWY6gun",
    "outputId": "c7c4e1f8-8b94-4131-b039-c85f53e2ce52"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\n",
       "  <div id=\"df-305dbb7f-8094-4c96-8856-74f5dd2fdd95\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>context</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...</td>\n",
       "      <td>差评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...</td>\n",
       "      <td>好评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...</td>\n",
       "      <td>好评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...</td>\n",
       "      <td>差评</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...</td>\n",
       "      <td>好评</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-305dbb7f-8094-4c96-8856-74f5dd2fdd95')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "\n",
       "\n",
       "\n",
       "    <div id=\"df-508b6443-10a5-48e9-aadd-70b20268551c\">\n",
       "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-508b6443-10a5-48e9-aadd-70b20268551c')\"\n",
       "              title=\"Suggest charts.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "      </button>\n",
       "    </div>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "    background-color: #E8F0FE;\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: #1967D2;\n",
       "    height: 32px;\n",
       "    padding: 0 0 0 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: #E2EBFA;\n",
       "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: #174EA6;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "    background-color: #3B4455;\n",
       "    fill: #D2E3FC;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart:hover {\n",
       "    background-color: #434B5C;\n",
       "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "    fill: #FFFFFF;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "    <script>\n",
       "      async function quickchart(key) {\n",
       "        const containerElement = document.querySelector('#' + key);\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      }\n",
       "    </script>\n",
       "\n",
       "      <script>\n",
       "\n",
       "function displayQuickchartButton(domScope) {\n",
       "  let quickchartButtonEl =\n",
       "    domScope.querySelector('#df-508b6443-10a5-48e9-aadd-70b20268551c button.colab-df-quickchart');\n",
       "  quickchartButtonEl.style.display =\n",
       "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "}\n",
       "\n",
       "        displayQuickchartButton(document);\n",
       "      </script>\n",
       "      <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-305dbb7f-8094-4c96-8856-74f5dd2fdd95 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-305dbb7f-8094-4c96-8856-74f5dd2fdd95');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "                                             context target\n",
       "0  [Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...     差评\n",
       "1  [Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...     好评\n",
       "2  [Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...     好评\n",
       "3  [Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...     差评\n",
       "4  [Round 1]\\n\\n问：文本分类任务：将一段用户给外卖服务的评论进行分类，分成好评或者...     好评"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dftrain['context'] = [build_inputs(x,history=his) for x in dftrain['text']] # 定义训练集中的上文\n",
    "dftrain['target'] = [x for x in dftrain['tag']] # 定义训练集中的标签\n",
    "dftrain = dftrain[['context','target']]\n",
    "\n",
    "dftest['context'] = [build_inputs(x,history=his) for x in dftest['text']]\n",
    "dftest['target'] = [x for x in dftest['tag']]\n",
    "dftest = dftest[['context','target']]\n",
    "\n",
    "# dftest.head()\n",
    "\n",
    "# dftrain.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "id": "mnvthqus7fBD"
   },
   "outputs": [],
   "source": [
    "# 将pandas的dataset转化为hf的dataset\n",
    "ds_train = datasets.Dataset.from_pandas(dftrain)\n",
    "ds_val = datasets.Dataset.from_pandas(dftest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "j2f6Hpbn-2fV",
    "outputId": "4074b633-f06c-432a-8866-7e5058482387"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['context', 'target'],\n",
       "    num_rows: 6000\n",
       "})"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j1ckBS-BZI3R"
   },
   "source": [
    "## 将数据进行 Tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "id": "A34JsLAV8N2H"
   },
   "outputs": [],
   "source": [
    "# 将输入的文本转化为token也就是Tokenizer\n",
    "# context转化成context_ids，把target转化成target_ids\n",
    "# 将context_ids和target_ids拼接到一起作为模型的input_ids\n",
    "\n",
    "from tqdm import tqdm\n",
    "import transformers\n",
    "\n",
    "model_name = \"THUDM/chatglm2-6b\"\n",
    "max_seq_length = 512\n",
    "skip_over_length = True\n",
    "\n",
    "# 载入模型的Tokenizer\n",
    "tokenizer = transformers.AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)\n",
    "# 载入模型的参数\n",
    "config = transformers.AutoConfig.from_pretrained(model_name, trust_remote_code=True, device_map='auto')\n",
    "# 定义预处理流程\n",
    "def preprocess(example):\n",
    "    context = example[\"context\"]\n",
    "    target = example[\"target\"]\n",
    "    # 将context转化为id 超长就截取\n",
    "    context_ids = tokenizer.encode(\n",
    "            context,\n",
    "            max_length=max_seq_length,\n",
    "            truncation=True)\n",
    "    # 将target也转化为id\n",
    "    target_ids = tokenizer.encode(\n",
    "        target,\n",
    "        max_length=max_seq_length,\n",
    "        truncation=True,\n",
    "        add_special_tokens=False)\n",
    "    # 将上述两者的id拼接 + 配置文件中end of sentence 的id\n",
    "    input_ids = context_ids + target_ids + [config.eos_token_id]\n",
    "\n",
    "    return {\"input_ids\": input_ids, \"context_len\": len(context_ids),'target_len':len(target_ids)}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 17,
     "referenced_widgets": [
      "61f8325c92ff4b9e9646d4593ff9aeeb",
      "1992b8e9af604d4684423094d2b625b0",
      "fc08cc8ada404964afc78d14d362a307",
      "0823b03adb944aaf867be87e1b32fc95",
      "4d56934947ec47b7a7ca161339ccd8a7",
      "36202730c3b940bc9b741fa2c98cd6d8",
      "e478d3c597d7447a9bc144f781919124",
      "4b58fd1b50804d16a9c723228e6bf9f5",
      "b00a3042be6e4462a982072b10f33d06",
      "f719fc7d187248aaab5616d12048ccc9",
      "40115d27540748a29589f8d5cd903bbc",
      "d2fdaf73aab94e0b9d61a24310a9eccc",
      "c603c024236f4f98a2e16402e08b61e0",
      "a874214dde164270a43c60f0bfee959e",
      "8036903c231f4da594e4735ef12e4e9b",
      "aa12dfc3bdbe4c79b2164d1b1f8c65c9",
      "9bc3d35d44864ae4b32fb4387492af30",
      "8fb6b350d38941038fe7a10673c42f49",
      "022ee630affe4d909ffd5d8c839f6252",
      "9b5410c75f4e49ec926f4d37528321b2",
      "451df9ca78c7409ab6bfd8891008d94a",
      "81afe9b91f5a4cbea3e1fd5563bb509e"
     ]
    },
    "id": "fshQ59uV_U3E",
    "outputId": "13ff96b0-f127-4ed9-f219-1eb6983dccd1"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "61f8325c92ff4b9e9646d4593ff9aeeb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/6000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d2fdaf73aab94e0b9d61a24310a9eccc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Filter:   0%|          | 0/6000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 先进行preprocess流程，然后基于该流程的结果选择三个列\n",
    "ds_train_token = ds_train.map(preprocess).select_columns(['input_ids', 'context_len','target_len'])\n",
    "if skip_over_length: # 基于跳过超长输入内容的条件，将训练集中太长的数据进行剔除\n",
    "    ds_train_token = ds_train_token.filter(\n",
    "        lambda example: example[\"context_len\"]<max_seq_length and example[\"target_len\"]<max_seq_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 17,
     "referenced_widgets": [
      "c38dba11819c4531ad37f023c427620e",
      "33207fc2cc5f4b6da3d9ac8f916a4a7d",
      "99e22f30c52847aa97bf1d31c46a29eb",
      "9d1e00e2347e42d0b8704e562cd51da6",
      "a6b108bd8a734774992f7cb89e02a9c6",
      "59acbc5323cd439b9b16a8df96b42fff",
      "8f076715836f4d0dbb6e082a84178d2b",
      "321002d3fcc849579fa3693f41d809a0",
      "85cd0c14acf440b2b426fec7ec8bf6a7",
      "08107c31b06742d2abb80bf3e0626a61",
      "be14f527c0374ca89def84c3cb778c4d",
      "f4d3233d514841739d1b5cad0371b0cc",
      "85fec979409f4e1c948ef76d1715cd85",
      "89f667e3508c4691a770c1325e80b9aa",
      "0ac5f7dab5324200be16659b1e950ca4",
      "f05cfbd11828456096260430e33e1df0",
      "d1feb1ad37024a33a209fad97394a4ba",
      "f4ae29eb498a4db1bc8c529ade45929e",
      "c9065d85291a46148eadb8faf3b8d088",
      "e638631281654e50b855a5daa7ee7c99",
      "c6de6497352f4e6c99e2c24a9a2ffeab",
      "07d7d5c813544d6c99694f3d903b78a4"
     ]
    },
    "id": "snYrcl9yALTS",
    "outputId": "33d924d7-6be8-4a9f-f8ee-d11758fc898b"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c38dba11819c4531ad37f023c427620e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/2000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f4d3233d514841739d1b5cad0371b0cc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Filter:   0%|          | 0/2000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 逻辑和上面一致，对验证集的处理\n",
    "ds_val_token = ds_val.map(preprocess).select_columns(['input_ids', 'context_len','target_len'])\n",
    "if skip_over_length:\n",
    "    ds_val_token = ds_val_token.filter(\n",
    "        lambda example: example[\"context_len\"]<max_seq_length and example[\"target_len\"]<max_seq_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "veOHKqUBZR5c"
   },
   "source": [
    "## 构建训练管道数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "id": "zpIzCJwtAoQ4"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "# 把一个batch的数据整理成适合模型训练的格式\n",
    "# 这里需要注意的是训练的思路，输入的信息是上下文 + 结论 + 句号 + padding，但是做loss的部分只有结论的部分 + 句号的部分也就是 target_id + eos_id\n",
    "def data_collator(features: list):\n",
    "    len_ids = [len(feature[\"input_ids\"]) for feature in features]\n",
    "    # 计算出这个batch中的所有样本的 input_ids 长度，并找出最长的长度\n",
    "    # 之后按照batch中最长的input_ids进行padding，不足的就用空补全\n",
    "    longest = max(len_ids)\n",
    "    # 初始化两个空列表 input_ids 和 labels_list，它们用于存储预处理后的输入数据和标签数据\n",
    "    input_ids = []\n",
    "    labels_list = []\n",
    "    # 对 len_ids 和 features 进行合并，然后按照 input_ids 的长度进行逆序排序，也就是说，最长的 input_ids 会排在前面\n",
    "    for length, feature in sorted(zip(len_ids, features), key=lambda x: -x[0]):\n",
    "        # 提取出当前样本的 input_ids 和 context_len\n",
    "        ids = feature[\"input_ids\"]\n",
    "        context_len = feature[\"context_len\"]\n",
    "        # 生成标签数据\n",
    "        # -100 是一个特殊的标记，它表示该位置的损失会在训练时被忽略\n",
    "        # 标签数据的生成规则：先放入 context_len  个 -100，然后放入 ids 从 context_len 开始到最后的部分，最后再放入 longest - length 个 -100\n",
    "        labels = (\n",
    "            [-100] * context_len + ids[context_len :] + [-100] * (longest - length)\n",
    "        ) #-100标志位后面会在计算loss时会被忽略不贡献损失，我们集中优化target部分生成的loss\n",
    "        # 对 input_ids 进行padding，如果 input_ids 的长度小于最长长度，那么在其后面添加足够数量的 pad_token_id\n",
    "        ids = ids + [tokenizer.pad_token_id] * (longest - length)\n",
    "        # 将处理过的 input_ids 和 labels 转为 LongTensor 类型，然后添加到相应的列表中\n",
    "        input_ids.append(torch.LongTensor(ids))\n",
    "        labels_list.append(torch.LongTensor(labels))\n",
    "\n",
    "    # 将 input_ids 和 labels 的列表堆叠为一个新的tensor\n",
    "    input_ids = torch.stack(input_ids)\n",
    "    labels = torch.stack(labels_list)\n",
    "    return {\n",
    "        \"input_ids\": input_ids,\n",
    "        \"labels\": labels,\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "id": "c1lc75j4C6Eo"
   },
   "outputs": [],
   "source": [
    "# ds_train_token 是送入训练的数据集\n",
    "# num_workers 是数据载入时将使用多线程并行处理，这可以在一定程度上加速数据载入\n",
    "# batch_size 是每一个批次的样本数量\n",
    "# pin_memory=True: 如果设为 True，那么数据载入器将会在返回Tensor之前，先将其复制到CUDA固定内存中。这样可以使得转移数据到GPU上更快\n",
    "# shuffle=True: 如果设为 True，那么在每个训练周期开始时，数据载入器将会打乱数据集的顺序\n",
    "# collate_fn=data_collator: 这个函数定义了如何将多个样本合并成一个小批量。在这里，我们使用之前定义的 data_collator 函数，这个函数会按照我们的需要对每个小批量的数据进行预处理\n",
    "dl_train = torch.utils.data.DataLoader(ds_train_token,num_workers=2,batch_size=4,\n",
    "                                       pin_memory=True,shuffle=True,\n",
    "                                       collate_fn = data_collator)\n",
    "dl_val = torch.utils.data.DataLoader(ds_val_token,num_workers=2,batch_size=4,\n",
    "                                    pin_memory=True,shuffle=True,\n",
    "                                     collate_fn = data_collator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "id": "s3Kn9ZJ2EzRh"
   },
   "outputs": [],
   "source": [
    "dl_train.size = 300 #用约300个step做一次验证"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UuZc8UtjF42y",
    "outputId": "0903cc0e-9d72-4ed2-feeb-f4e1723b0f3b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thu Jul 13 02:34:06 2023       \n",
      "+-----------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |\n",
      "|-------------------------------+----------------------+----------------------+\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|                               |                      |               MIG M. |\n",
      "|===============================+======================+======================|\n",
      "|   0  NVIDIA A100-SXM...  Off  | 00000000:00:04.0 Off |                    0 |\n",
      "| N/A   34C    P0    49W / 400W |  12755MiB / 40960MiB |      0%      Default |\n",
      "|                               |                      |             Disabled |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "                                                                               \n",
      "+-----------------------------------------------------------------------------+\n",
      "| Processes:                                                                  |\n",
      "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
      "|        ID   ID                                                   Usage      |\n",
      "|=============================================================================|\n",
      "+-----------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "import locale\n",
    "locale.getpreferredencoding = lambda: \"UTF-8\"\n",
    "# 检查设备后开始训练\n",
    "# for colab tricky issue\n",
    "# https://github.com/googlecolab/colabtools/issues/3409\n",
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wYbfmIEIZX7d"
   },
   "source": [
    "## 定义模型：Lora+ChatGLM2-6B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "id": "CVKkEE-QGcCA"
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67,
     "referenced_widgets": [
      "420800e9429b45ee9b48f2d81388025a",
      "a0eb595e5fb1445a86a4fc4724196c60",
      "687484ad16da42a986ca785667ace01c",
      "b9d272b52bb149d8b07d32bcc7ee6b1a",
      "7cd70c7b987147a8a65f42cf25e4578b",
      "35da59a976ab401381793a70b087f849",
      "cba1a27cd05340f191e9093a0098c6f7",
      "a70f769b20a6426dbf54f18d3df01089",
      "8acd915a527b47188d1158f4bbb010e4",
      "cf48a73ae8034620a8c946e92724081f",
      "e0da03b6f9b943e69095c5fba3a582f3"
     ]
    },
    "id": "ENd5D7dkGhUf",
    "outputId": "54655151-09fe-4c27-e778-3f5c6a72dc5b"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "420800e9429b45ee9b48f2d81388025a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/7 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trainable params: 1949696 || all params: 6245533696 || trainable%: 0.031217444255383614\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoTokenizer, AutoModel, TrainingArguments, AutoConfig\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from peft import get_peft_model, LoraConfig, TaskType\n",
    "\n",
    "# 加载ChatGLM2-6B模型并不以8bit的精度载入\n",
    "model = AutoModel.from_pretrained(\"THUDM/chatglm2-6b\",\n",
    "                                  load_in_8bit=False,\n",
    "                                  trust_remote_code=True)\n",
    "\n",
    "model.supports_gradient_checkpointing = True  #节约cuda，但可能会使得训练时间变长\n",
    "model.gradient_checkpointing_enable() # 作用同上\n",
    "model.enable_input_require_grads() # 作用同上\n",
    "\n",
    "model.config.use_cache = False  # 关闭了模型的缓存机制，该设置可以避免一些警告，但在模型推理时需要重新开启\n",
    "\n",
    "# 配置LORA模型的类\n",
    "# 常规LM任务\n",
    "# 非推理模式\n",
    "peft_config = LoraConfig(\n",
    "    task_type=TaskType.CAUSAL_LM, inference_mode=False,\n",
    "    r=8,\n",
    "    lora_alpha=32, lora_dropout=0.1,\n",
    ")\n",
    "\n",
    "# 结合Lora和原有模型\n",
    "model = get_peft_model(model, peft_config)\n",
    "# 开启模型的并行处理能力，这可以在有多个GPU的情况下提高训练效率\n",
    "model.is_parallelizable = True\n",
    "model.model_parallel = True\n",
    "# 打印出模型的可训练参数\n",
    "model.print_trainable_parameters()\n",
    "# 可训练参数：1949696\n",
    "# 总参数量：6245533696\n",
    "# 需要调整的模型参数量的占比还是很低的 3.1%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IRSEuv6vZf2q"
   },
   "source": [
    "## 训练模型的流程"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "id": "wpe2Kl5yIJPg"
   },
   "outputs": [],
   "source": [
    "from torchkeras import KerasModel\n",
    "from accelerate import Accelerator\n",
    "# 重写了torchkeras中的StepRunner类\n",
    "class StepRunner:\n",
    "    def __init__(self, net, loss_fn, accelerator=None, stage = \"train\", metrics_dict = None,\n",
    "                 optimizer = None, lr_scheduler = None\n",
    "                 ):\n",
    "        self.net,self.loss_fn,self.metrics_dict,self.stage = net,loss_fn,metrics_dict,stage\n",
    "        self.optimizer,self.lr_scheduler = optimizer,lr_scheduler\n",
    "        self.accelerator = accelerator if accelerator is not None else Accelerator()\n",
    "        if self.stage=='train':\n",
    "            self.net.train()\n",
    "        else:\n",
    "            self.net.eval()\n",
    "\n",
    "    def __call__(self, batch):\n",
    "\n",
    "        # 计算loss\n",
    "        with self.accelerator.autocast():\n",
    "            # 通过模型self.net对输入数据进行预测，然后用预测结果和标签计算损失, loss的计算过程直接交给ChatGLM2了吗？\n",
    "            loss = self.net(input_ids=batch[\"input_ids\"],labels=batch[\"labels\"]).loss\n",
    "\n",
    "        # 执行梯度下降的步骤\n",
    "        if self.optimizer is not None and self.stage==\"train\":\n",
    "            # 计算每个模型参数关于损失的梯度\n",
    "            self.accelerator.backward(loss)\n",
    "            # 执行梯度裁剪（self.accelerator.clip_grad_norm_），防止梯度爆炸\n",
    "            if self.accelerator.sync_gradients:\n",
    "                self.accelerator.clip_grad_norm_(self.net.parameters(), 1.0)\n",
    "            # 更新模型参数\n",
    "            self.optimizer.step()\n",
    "            # 存在学习率调整器，就调整学习率\n",
    "            if self.lr_scheduler is not None:\n",
    "                self.lr_scheduler.step()\n",
    "            # 清空梯度\n",
    "            self.optimizer.zero_grad()\n",
    "\n",
    "        # 给并行计算和多GPU计算的损失收集起来并求和\n",
    "        all_loss = self.accelerator.gather(loss).sum()\n",
    "\n",
    "        # losses (or plain metrics that can be averaged)\n",
    "        step_losses = {self.stage+\"_loss\":all_loss.item()}\n",
    "\n",
    "        # metrics (stateful metrics)\n",
    "        step_metrics = {}\n",
    "\n",
    "        # 记录过程信息，追踪模型性能用\n",
    "        if self.stage==\"train\":\n",
    "            if self.optimizer is not None:\n",
    "                step_metrics['lr'] = self.optimizer.state_dict()['param_groups'][0]['lr']\n",
    "            else:\n",
    "                step_metrics['lr'] = 0.0\n",
    "        return step_losses,step_metrics\n",
    "\n",
    "KerasModel.StepRunner = StepRunner\n",
    "\n",
    "# 仅仅保存lora可训练参数\n",
    "# 覆盖了KerasModel中的load_ckpt和save_ckpt方法\n",
    "def save_ckpt(self, ckpt_path='checkpoint', accelerator = None):\n",
    "    unwrap_net = accelerator.unwrap_model(self.net)\n",
    "    unwrap_net.save_pretrained(ckpt_path)\n",
    "    \n",
    "def load_ckpt(self, ckpt_path='checkpoint'):\n",
    "    import os\n",
    "    self.net.load_state_dict(\n",
    "        torch.load(os.path.join(ckpt_path,'adapter_model.bin')),strict =False)\n",
    "    self.from_scratch = False\n",
    "\n",
    "KerasModel.save_ckpt = save_ckpt\n",
    "KerasModel.load_ckpt = load_ckpt\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "id": "7wA8VypDIbO2"
   },
   "outputs": [],
   "source": [
    "# 定义模型的训练\n",
    "# 没有损失函数\n",
    "# 优化器是adamW\n",
    "# 学习率 2e-6\n",
    "# 模型参数存储路径\n",
    "keras_model = KerasModel(model,loss_fn = None,\n",
    "        optimizer=torch.optim.AdamW(model.parameters(),lr=2e-6))\n",
    "ckpt_path = 'waimai_chatglm2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "LywFMbSKIh5f",
    "outputId": "b483f05c-b5d2-43a5-cafb-9ba0f45f6d51"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< ⚡️ cuda is used >>>>>>\u001b[0m\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiEAAAGJCAYAAABcsOOZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABvBElEQVR4nO3dd3xTVf8H8M9t2qZ7bygUKLOWVoYVkKGUqWxkKkvlEQFBhoiylEeWiGxQH1kqCLL8KUOhUkS2lG0FioUyuigdtKVNm5zfH5cGQtOdNB2f9+t1X01uzj33nNym+fasKwkhBIiIiIjKmZmpC0BERETVE4MQIiIiMgkGIURERGQSDEKIiIjIJBiEEBERkUkwCCEiIiKTYBBCREREJsEghIiIiEyCQQgRERGZBIMQMro5c+ZAkiTcu3fP1EUpNzdu3IAkSdiwYYNRj6HKZ9GiRWjUqBE0Go2pi1JuKuLfgLVr16JWrVrIzs42dVGqNQYhVGXNmzcPu3fvNnUx6CkpKSkYPXo03N3dYWtrixdffBERERHFPj4yMhJdu3aFnZ0dXFxc8PrrryMxMVFv2uvXr2PIkCHw8PCAtbU16tevj48++kgnzYgRIyBJUr6tUaNGOunu3r2L1157DQ0bNoS9vT2cnJzw3HPPYePGjSju3S/S0tKwcOFCTJs2DWZmRf/5zcrKQk5OTrHyrsquXLmC9957D61bt4aVlRUkScKNGzf0pn3vvffQrFkzuLi4wMbGBo0bN8acOXOQnp6uk27EiBFQqVT48ssvy6EGVBBzUxeAyFjmzZuH/v37o3fv3qYuCj2i0Wjw8ssv4/z585g6dSrc3NywevVqdOjQAWfOnEH9+vULPf727dto164dHB0dMW/ePKSnp2Px4sW4ePEiTp06BUtLS23ac+fOoUOHDqhRowYmT54MV1dXxMTE4NatW/nyVSqV+N///qezz9HRUef5vXv3cPv2bfTv3x+1atVCTk4ODhw4gBEjRuDKlSuYN29ekfVft24dcnNzMXjw4ALTnD59GsuXL8dvv/2GhIQESJKEGjVqoE+fPnj33Xfh7+9f5HmqmuPHj2P58uVo0qQJGjdujHPnzhWY9vTp02jbti1GjhwJKysrnD17FgsWLMDBgwfxxx9/aIM/KysrDB8+HEuWLMH48eMhSVI51YZ0CCIjmz17tgAgEhMTy/W8tra2Yvjw4eV6zjzR0dECgFi/fr1Rj6lstm7dKgCIH3/8UbsvISFBODk5icGDBxd5/JgxY4S1tbW4efOmdt+BAwcEAPHll19q96nVavHMM8+IkJAQkZmZWWiew4cPF7a2tqWojeyVV14Rtra2Ijc3t8i0TZs2Fa+99pre13JycsTYsWOFJEmibdu2YvHixeLnn38WO3bsEPPmzRPBwcHCyspKrFy5stRlNZWy/g1ISkoSaWlpQgghPvvsMwFAREdHF/v4xYsXCwDi+PHjOvv/+usvAUCEhYWVqlxUduyOoXJz7949DBgwAA4ODnB1dcWECROQlZWVL913332H5s2bw9raGi4uLhg0aFC+/16vXbuGfv36wcvLC1ZWVqhZsyYGDRqE1NRUAIAkScjIyMDGjRu1zesjRozQW674+HiYm5vj448/zvfalStXIEkSVq5cCQC4f/8+pkyZgsDAQNjZ2cHBwQHdunXD+fPny/juFOz3339H27ZtYWtrCycnJ/Tq1QuRkZE6aR48eICJEyfCz88PSqUSHh4e6NSpk043R1HvWXnYvn07PD090bdvX+0+d3d3DBgwAD/99FOR/fM7duzAK6+8glq1amn3hYaGokGDBti2bZt232+//YZLly5h9uzZsLa2RmZmJtRqdaF5q9VqpKWllbhOfn5+yMzMhEqlKjRddHQ0Lly4gNDQUL2vjxw5Eps3b8bevXvxxx9/YPLkyXjllVfQt29fTJ8+HWfPnsXatWsxZcoUrF27Nt/xd+7cwahRo+Dp6QmlUomAgACsW7dOJ014eDgkScLWrVvx4YcfwsvLC7a2tujZs6feFqIff/xR+1l0c3PDa6+9hjt37uRL988//2DAgAFwd3eHtbU1GjZsmK/bC5C74kaMGAEnJyc4Ojpi5MiRyMzMLPR9AwAXFxfY29sXma4gfn5+2vM/qXnz5nBxccFPP/1U6rypbNgdQ+VmwIAB8PPzw/z583HixAksX74cycnJ2LRpkzbNp59+ipkzZ2LAgAF48803kZiYiBUrVqBdu3Y4e/YsnJycoFKp0KVLF2RnZ2P8+PHw8vLCnTt38MsvvyAlJQWOjo749ttv8eabb+K5557D6NGjAQD16tXTWy5PT0+0b98e27Ztw+zZs3Ve27p1KxQKBV599VUAwL///ovdu3fj1VdfRZ06dRAfH48vv/wS7du3x99//w0fHx+DvmcHDx5Et27dULduXcyZMwcPHz7EihUr0KZNG0RERGj/uL799tvYvn07xo0bhyZNmiApKQl//vknIiMj0axZs2K9ZwXJzMws1heFQqGAs7NzoWnOnj2LZs2a5RsP8dxzz+Grr77C1atXERgYqPfYO3fuICEhAS1atMj32nPPPYe9e/dqnx88eBCA3M3SokULnDlzBpaWlujTpw9Wr14NFxeXfHV0cHBAZmYmnJ2dMXjwYCxcuBB2dnb5zvXw4UNkZGQgPT0dhw8fxvr169GqVStYW1sXWvdjx44BAJo1a5bvtW+//Ra7du3CyZMnERAQAAAQQiAjI0Nbhnv37uH111+Hm5sbXn31VXTr1g21a9cGIAfSzz//PCRJwrhx4+Du7o59+/bhjTfeQFpaGiZOnKhzvk8//RSSJGHatGlISEjA0qVLERoainPnzmnrsWHDBowcORItW7bE/PnzER8fj2XLluHo0aPazyIAXLhwAW3btoWFhQVGjx4NPz8/XL9+HT///DM+/fRTnfMOGDAAderUwfz58xEREYH//e9/8PDwwMKFCwt970oqNzcXKSkpUKlUuHTpEmbMmAF7e3s899xz+dI2a9YMR48eNej5qQRM3RRDVV9eU2zPnj119r/zzjsCgDh//rwQQogbN24IhUIhPv30U510Fy9eFObm5tr9Z8+ezdekr09JumO+/PJLAUBcvHhRZ3+TJk3ESy+9pH2elZUl1Gq1Tpro6GihVCrFJ598orMPBuiOCQ4OFh4eHiIpKUm77/z588LMzEwMGzZMu8/R0VGMHTu2wLyL+57pk3f9itpq165dZF62trZi1KhR+fbv2bNHABD79+8v8NjTp08LAGLTpk35Xps6daoAILKysoQQQvTs2VMAEK6urmLo0KFi+/btYubMmcLc3Fy0bt1aaDQa7bEffPCBmDZtmti6davYsmWLGD58uAAg2rRpI3JycvKda/78+Tr17tixo4iJiSmy7jNmzBAAxIMHD3T2azQaUadOHbF06VLtvp9++kn4+PgIAKJWrVri119/1emC6NOnj/jwww+16d944w3h7e0t7t27p5P3oEGDhKOjo7ZL6tChQwKAqFGjhrZ7Qwghtm3bJgCIZcuWCSGEUKlUwsPDQzzzzDPi4cOH2nS//PKLACBmzZql3deuXTthb2+v00WWV688eb9DT1/7Pn36CFdX1yLfuycVpzvm+PHjOteoYcOG4tChQ3rTjh49WlhbW5eoDGQ4bAmhcjN27Fid5+PHj8fq1auxd+9eNG3aFDt37oRGo8GAAQN0pvJ5eXmhfv36OHToED788EPtf+2//vorunfvDhsbmzKXrW/fvhg7diy2bt2KZ555BgBw6dIl/P3335gwYYI2nVKp1D5Wq9VISUmBnZ0dGjZsWKIZHsURGxuLc+fO4f3339f5z71p06bo1KmTzn/+Tk5OOHnyJO7evau3NaYs79mwYcPwwgsvFJmuqJYAQG5FePI9zGNlZaV9vbBjARR5vFKp1M6EaNmyJb777jsAQL9+/WBjY4Pp06cjLCxM2y0yf/58nbwGDRqEBg0a4KOPPsL27dsxaNAgndcHDx6MFi1aIDExEb/88gvi4+MLLXeepKQkmJub52tdOXPmDBISEvDGG28AkFt8Bg8ejM6dO+P1119HTEwMRo0apXNM7969tbM6hBDYsWMHBgwYACGEzmenS5cu+OGHHxAREYE2bdpo9w8bNkyne6N///7w9vbG3r178e677+Kvv/5CQkIC5syZo31vAeDll19Go0aNsGfPHnz88cdITEzEH3/8gQkTJuh0kQHQO9Dz7bff1nnetm1b7Nq1C2lpaXBwcCjyPSyuJk2a4MCBA8jIyMCxY8dw8ODBfLNj8jg7O+Phw4fIzMw0yN8SKhkGIVRunp75UK9ePZiZmWmn2l27dg1CiAJnSFhYWAAA6tSpg0mTJmHJkiX4/vvv0bZtW/Ts2ROvvfZaod0KhXFzc0PHjh2xbds2zJ07F4DcFWNubq4zfkGj0WDZsmVYvXo1oqOjdcYZuLq6lurcBbl58yYAoGHDhvlea9y4MX799VdkZGTA1tYWixYtwvDhw+Hr64vmzZuje/fuGDZsGOrWrQugbO9Z3bp1tfmUlbW1td5xH3ljgwoLZPJeK87xeT+fnoUyZMgQTJ8+HceOHStwbAYgT/OcOXMmDh48mC8IqV27trYbZPDgwRg9ejRCQ0Nx5cqVYgViTztz5gxatGihDU6+//571KhRA9u3b4dCoQAgB5kjR47UHuPp6amdlpyYmIiUlBR89dVX+Oqrr/SeIyEhQef5058xSZLg7++v/SwW9rvXqFEj/PnnnwDk7kkA2sC9KE8HKnndd8nJyQYNQhwcHLTXt1evXti8eTN69eqFiIgIBAUF6aQVj6ZXc3aMaXBgKpnM0x96jUYDSZKwf/9+HDhwIN/25Hz+zz//HBcuXMCHH36Ihw8f4t1330VAQABu375d6vIMGjQIV69e1U7/27ZtGzp27Ag3Nzdtmnnz5mHSpElo164dvvvuO/z66684cOAAAgICTLr41IABA/Dvv/9ixYoV8PHxwWeffYaAgADs27dPm6a071l6ejri4uKK3Apaq+NJ3t7eiI2Nzbc/b19hY2q8vb110j59vIuLi7aVJC8fT09PnXQeHh4A5C+9wlhbW8PV1RX3798vNB0gtyLcunULf/zxR6HpXF1dkZubiwcPHujsT0pK0qn3jRs38Oyzz2oDEAD5xjLcunVLG/Tm/d699tprej83Bw4c0GkFMaUn6/QkUcx1Vkor7x+JH374Id9rycnJsLGxKVUASWXHlhAqN9euXUOdOnW0z6OioqDRaLSDK+vVqwchBOrUqYMGDRoUmV9gYCACAwMxY8YMHDt2DG3atMHatWvx3//+F0DJ/7Pp3bs3/vOf/2Dr1q0AgKtXr2L69Ok6abZv344XX3wR33zzjc7+lJQUnWDFEPL+275y5Uq+1/755x+4ubnB1tZWu8/b2xvvvPMO3nnnHSQkJKBZs2b49NNP0a1bN22aot4zfRYvXqx35pC+8ha0gFSe4OBgHDlyBBqNRmdw6smTJ2FjY1Poda9Rowbc3d3x119/5Xvt1KlTCA4O1j5v3rw5vv7663wzOe7evQtAnpFTmAcPHuDevXtFpgMedxMVNcsob/Gz6OhoNG3aVLvfwcFB51gvLy+cOnVK59i8FgdA/sL+5ptvtP/pu7u7w97eHmq1utDWnSddu3ZN57kQAlFRUdpyPfm799JLL+mkvXLlivb1vBayS5cuFeu8ppKdnQ2NRqP3GkVHR6Nx48YmKBUBbAmhcrRq1Sqd5ytWrAAA7Zdk3759oVAo8PHHH+f7z0gIgaSkJADyqpO5ubk6rwcGBsLMzEynqd7W1jbflLzCODk5oUuXLti2bRt++OEHWFpa5lvoTKFQ5Cvbjz/+qHfaYll5e3sjODgYGzdu1KnHpUuX8Ntvv6F79+4A5LEpT/9x9fDwgI+Pj/b9KO57ps+wYcMK/A/7ye37778vsk79+/dHfHw8du7cqd137949/Pjjj+jRo4fOeI/r16/j+vXrOsf369cPv/zyi8500rCwMFy9elU7gwmQm+CVSiXWr1+v00KVtyBZp06dAMjdOE+3TADA3LlzIYRA165dtfsKaun55ptvIEmS3lkvT2rVqhUA5AuiGjdujNOnT2vL2atXL5w9exazZs3Cv//+iyNHjmDq1KkA5NlF/fr1w+3bt7VjlRQKBfr164cdO3boDQb0lXvTpk069d6+fTtiY2O1n8UWLVrAw8MDa9eu1fn92LdvHyIjI/Hyyy8DkAOgdu3aYd26dYiJidE5h7FbN/RJSUnRu8Js3nXXN7MqIiICrVu3NnrZqAAmGQ5L1UreyPjAwEDRo0cPsWrVKvHaa68JAGLIkCE6afNmHrRu3VosWrRIrFmzRrz//vuifv364rPPPhNCCLFr1y5Ro0YNMXHiRLF69WqxfPly0bJlS2FhYaGzGFH37t2Fra2t+Pzzz8WWLVvEiRMniizrd999JwAIe3t70aNHj3yvz5o1SwAQI0aMEF999ZUYP368cHFxEXXr1hXt27fXpjPU7JgDBw4Ic3Nz0ahRI/HZZ5+JTz75RLi7uwtnZ2fx77//CiGESE5O1s4EWrJkifjqq6/EgAEDBADx+eefl+g9M7bc3Fzx/PPPCzs7O/Hxxx+LVatWiYCAAGFvby/++ecfnbS1a9fON+MmJiZGuLq6inr16only5eLefPmCWdnZxEYGKidGZPnk08+EQBEp06dxKpVq8To0aOFJEk6i6JFR0cLJycnMWbMGLFs2TKxbNky0b17dwFAdO3aVWcm1IQJE0SLFi3EjBkzxFdffSUWLFggWrZsKQCI8ePHF6v+zzzzTL5F2bKysoSjo6PYtWuXdt+8efOEmZmZACDMzc3FsmXLtDM9OnfurL32eeLi4kTt2rWFjY2NmDBhgvjyyy/F/PnzxauvviqcnZ216fJmxwQGBoqmTZuKL774QnzwwQfCyspK+Pv7i4yMDG3a9evXCwAiJCRELF26VEyfPl3Y2NgIPz8/kZycrE137tw5YWdnJ1xdXcX06dPFV199JT788EMRFBSkTVPQYmV55yhq4bGUlBQxd+5cMXfuXNG1a1cBQEyePFnMnTtXrFixQptu165dwtfXV7z33nti9erVYunSpaJfv35CkiTRokULkZ2drZNv3mJlBw8eLPT8ZDwMQsjo8v4A/f3336J///7C3t5eODs7i3HjxulM/8uzY8cO8cILLwhbW1tha2srGjVqJMaOHSuuXLkihBDi33//FaNGjRL16tUTVlZWwsXFRbz44ov5/pD8888/ol27dsLa2loAKNZ03bS0NG367777Lt/rWVlZYvLkycLb21tYW1uLNm3aiOPHj4v27dsbJQgRQoiDBw+KNm3aCGtra+Hg4CB69Ogh/v77b+3r2dnZYurUqSIoKEjY29sLW1tbERQUJFavXq1NU9z3rDzcv39fvPHGG8LV1VXY2NiI9u3bi9OnT+dLpy8IEUKIS5cuic6dOwsbGxvh5OQkhg4dKuLi4vKl02g0YsWKFaJBgwbCwsJC+Pr6ihkzZgiVSqVNk5ycLF577TXh7+8vbGxshFKpFAEBAWLevHk66YQQ4rfffhOvvPKK8PHxERYWFsLe3l60adNGrF+/Xmc6amGWLFki7Ozs8q3iOnv2bFG3bl1x//597b47d+6IP/74Q1u3P//8UyQkJBSYd3x8vBg7dqzw9fUVFhYWwsvLS3Ts2FF89dVX2jR5QciWLVvE9OnThYeHh7C2thYvv/xyvim2Qsgr3D777LNCqVQKFxcXMXToUHH79u186S5duiT69OkjnJychJWVlWjYsKGYOXOmTv3KEoTkfTb0bU/+jkRFRYlhw4aJunXrCmtra2FlZSUCAgLE7NmzRXp6er58p02bJmrVqlXs60eGJwlhgjYzIqJqKDU1FXXr1sWiRYu0U3IBuVuoTZs2UCgU+Omnn7SDcJ+2fft29OnTp8ABnkUJDw/Hiy++iB9//BH9+/cvVR5VRXZ2Nvz8/PDBBx/oTMOn8sUxIURE5cTR0RHvv/8+PvvsM52xKlZWVti7dy8kSULDhg0xbdo0/PHHH7h58yb++ecfbNq0Ca1atcLw4cMNvh5NdbV+/XpYWFjkW7uEyhdbQoiMTKVSFTnV09HRkVMECSqVCitXrsTKlSsRHR2t3W9lZYU+ffrg448/LvJOw4VhSwhVNJyiS2Rkx44dw4svvlhomvXr1xd4gz2qPiwtLTFp0iRMmjQJN27cwJ07d2BlZYXGjRtzNU+qktgSQmRkycnJOHPmTKFpAgICChwHQERUVTEIISIiIpPgwFQiIiIyCY4J0UOj0eDu3buwt7fnTY2IiIhKQAiBBw8ewMfHR+f2DPowCNHj7t278PX1NXUxiIiIKq1bt26hZs2ahaZhEKKHvb09APkNNOTtpYmIiKq6tLQ0+Pr6ar9LC8MgRI+8LhgHBwcGIURERKVQnOEMHJhKREREJsEghIiIiEyCQQgRERGZBMeEEBFRuRFCIDc3F2q12tRFoVJSKBQwNzc3yBIWDEKIiKhcqFQqxMbGIjMz09RFoTKysbGBt7c3LC0ty5QPgxAiIjI6jUaD6OhoKBQK+Pj4wNLSkotBVkJCCKhUKiQmJiI6Ohr169cvckGywjAIKQdqNXDkCBAbC3h7A23bAgqFqUtFRFR+VCoVNBoNfH19eUfgSs7a2hoWFha4efMmVCoVrKysSp0XgxAj27kTmDABuH378b6aNYFly4C+fU1XLiIiUyjLf81UcRjqOvK3wYh27gT699cNQADgzh15/86dpikXERFRRcAgxEjUarkFRIj8r+XtmzhRTkdERFQdMQgxkiNH8reAPEkI4NYtOR0RERWPWg2EhwNbtsg/K9s/cn5+fli6dKlB8goPD4ckSUhJSTFIfqbAMSFGEhtr2HRERNWdqcbYdejQAcHBwQYJHk6fPg1bW9uyF6qKYEuIkXh7GzYdEVF1VpHH2OUtwFYc7u7unB30BAYhRtK2rRyhFzQNXpIAX185HRFRdZaRUfCWlVW8MXYTJgDp6UXnW1IjRozA4cOHsWzZMkiSBEmSsGHDBkiShH379qF58+ZQKpX4888/cf36dfTq1Quenp6ws7NDy5YtcfDgQZ38nu6OkSQJ//vf/9CnTx/Y2Nigfv36+L//+7+SF/SRHTt2ICAgAEqlEn5+fvj88891Xl+9ejXq168PKysreHp6on///trXtm/fjsDAQFhbW8PV1RWhoaHIKM2bVgIMQoxEoZCbCIH8gUje86VLuV4IEZGdXcFbv37FG2N3+zbwwgu6+/388udXUsuWLUOrVq3w1ltvITY2FrGxsfD19QUAfPDBB1iwYAEiIyPRtGlTpKeno3v37ggLC8PZs2fRtWtX9OjRAzExMYWe4+OPP8aAAQNw4cIFdO/eHUOHDsX9+/dLXNYzZ85gwIABGDRoEC5evIg5c+Zg5syZ2LBhAwDgr7/+wrvvvotPPvkEV65cwf79+9GuXTsAQGxsLAYPHoxRo0YhMjIS4eHh6Nu3L4S+yM+QBOWTmpoqAIjU1NQy57VjhxA1awohf0zkzddX3k9EVF08fPhQ/P333+Lhw4f5Xnvy7+PTW/fuQmzeXHiavK1xY9183dzypymN9u3biwkTJmifHzp0SAAQu3fvLvLYgIAAsWLFCu3z2rVriy+++OKJukPMmDFD+zw9PV0AEPv27Ssy77xyJCcnCyGEGDJkiOjUqZNOmqlTp4omTZoIIYTYsWOHcHBwEGlpafnyOnPmjAAgbty4UeR5hSj8epbkO7RCtISsWrUKfn5+sLKyQkhICE6dOlVg2p07d6JFixZwcnKCra0tgoOD8e233+qkEUJg1qxZ8Pb2hrW1NUJDQ3Ht2jVjV0Ovvn2BGzeAn39+vO/CBS5URkSUJz294G3HjuKPnfviC93nN27kz8+QWrRoofM8PT0dU6ZMQePGjeHk5AQ7OztERkYW2RLStGlT7WNbW1s4ODggISGhxOWJjIxEmzZtdPa1adMG165dg1qtRqdOnVC7dm3UrVsXr7/+Or7//nvtfXyCgoLQsWNHBAYG4tVXX8XXX3+N5OTkEpehpEwehGzduhWTJk3C7NmzERERgaCgIHTp0qXAC+Di4oKPPvoIx48fx4ULFzBy5EiMHDkSv/76qzbNokWLsHz5cqxduxYnT56Era0tunTpgqysrPKqlg6FAnjlFcDNTX4eHW2SYhARVUi2tgVvVlbFH2MXGlp0voYtt26GU6ZMwa5duzBv3jwcOXIE586dQ2BgIFQqVaH5WFhY6DyXJAkajcawhQVgb2+PiIgIbNmyBd7e3pg1axaCgoKQkpIChUKBAwcOYN++fWjSpAlWrFiBhg0bItrIX1gmD0KWLFmCt956CyNHjkSTJk2wdu1a2NjYYN26dXrTd+jQAX369EHjxo1Rr149TJgwAU2bNsWff/4JQG4FWbp0KWbMmIFevXqhadOm2LRpE+7evYvdu3eXY83yW7JEHsFdp45Ji0FEVKmYeoydpaUl1MVYkOTo0aMYMWIE+vTpg8DAQHh5eeHGjRvGKZQejRs3xtGjR/OVqUGDBlA8enPMzc0RGhqKRYsW4cKFC7hx4wZ+//13AHLw06ZNG3z88cc4e/YsLC0tsWvXLqOW2aRBiEqlwpkzZxD6RPhqZmaG0NBQHD9+vMjjhRAICwvDlStXtINroqOjERcXp5Ono6MjQkJCCswzOzsbaWlpOpsxvP460KcP4ORklOyJiKqsvn2B7duBGjV099esKe83Zhe3n58fTp48iRs3buDevXsFtlLUr18fO3fuxLlz53D+/HkMGTLEKC0aBZk8eTLCwsIwd+5cXL16FRs3bsTKlSsxZcoUAMAvv/yC5cuX49y5c7h58yY2bdoEjUaDhg0b4uTJk5g3bx7++usvxMTEYOfOnUhMTETjxo2NWmaTBiH37t2DWq2Gp6enzn5PT0/ExcUVeFxqairs7OxgaWmJl19+GStWrECnTp0AQHtcSfKcP38+HB0dtVveyGciIqo48sbYHToEbN4s/4yONv4YuylTpkChUKBJkyZwd3cvcIzHkiVL4OzsjNatW6NHjx7o0qULmjVrZtzCPaFZs2bYtm0bfvjhBzzzzDOYNWsWPvnkE4wYMQIA4OTkhJ07d+Kll15C48aNsXbtWmzZsgUBAQFwcHDAH3/8ge7du6NBgwaYMWMGPv/8c3Tr1s2oZZaEMPb8m4LdvXsXNWrUwLFjx9CqVSvt/vfffx+HDx/GyZMn9R6n0Wjw77//Ij09XRv17d69Gx06dMCxY8fQpk0b3L17F95PjGYaMGAAJEnC1q1b8+WXnZ2N7Oxs7fO0tDT4+voiNTUVDg4OBqtvSor8ocnIAF57zWDZEhFVeFlZWYiOjkadOnXKdOt3qhgKu55paWlwdHQs1neoSZdtd3Nzg0KhQHx8vM7++Ph4eHl5FXicmZkZ/P39AQDBwcGIjIzE/Pnz0aFDB+1x8fHxOkFIfHw8goOD9eanVCqhVCrLWJui/fuvHLG7uzMIISIiMml3jKWlJZo3b46wsDDtPo1Gg7CwMJ2WkaJoNBptS0adOnXg5eWlk2daWhpOnjxZojyNoX59+WdiIpCaatKiEBFRBff222/Dzs5O7/b222+bungGYfIb2E2aNAnDhw9HixYt8Nxzz2Hp0qXIyMjAyJEjAQDDhg1DjRo1MH/+fADy+I0WLVqgXr16yM7Oxt69e/Htt99izZo1AOTRvRMnTsR///tf1K9fH3Xq1MHMmTPh4+OD3r17m6qaAAB7e8DTE4iPB6KigObNTVocIiKqwD755BPtoNKnGXKogCmZPAgZOHAgEhMTMWvWLMTFxSE4OBj79+/XDiyNiYmBmdnjBpuMjAy88847uH37NqytrdGoUSN89913GDhwoDbN+++/j4yMDIwePRopKSl44YUXsH///grRD+nvLwch164xCCEiooJ5eHjAw8PD1MUwKpMOTK2oSjKopqRGjgQ2bAA++QSYOdOgWRMRVVgcmFq1GGpgqskXK6tu8saFmGgVeSIiogqDQUg5YxBCREQkM/mYkOrmhReAH38EjLwIHRERUYXHIKSceXsD/fubuhRERESmx+4YIiKqNNRCIDw5GVvi4xGenAx1JZhb4efnh6VLlxYrrSRJJr/ZanliS4gJHD0KnDgBtGsHtGxp6tIQEVUOOxMTMSEqCrefuM1GTaUSy/z90dfd3YQlo9JiS4gJfPMNMGUKsG+fqUtCRFQ57ExMRP/Ll3UCEAC4k52N/pcvY2dioolKRmXBIMQEOEOGiAgQQiBDrS5yS8vNxbvXrkFfx0vevglRUUjLzS1WfiVZHuurr76Cj48PNBqNzv5evXph1KhRuH79Onr16gVPT0/Y2dmhZcuWOHjwYOnflKdcvHgRL730EqytreHq6orRo0cjPT1d+3p4eDiee+452NrawsnJCW3atMHNmzcBAOfPn8eLL74Ie3t7ODg4oHnz5vjrr78MVjZDYHeMCTAIISICMjUa2B05UuZ8BIDb2dlw/PPPYqVPb9sWtgpFsdK++uqrGD9+PA4dOoSOHTsCAO7fv4/9+/dj7969SE9PR/fu3fHpp59CqVRi06ZN6NGjB65cuYJatWqVtkoA5BXCu3TpglatWuH06dNISEjAm2++iXHjxmHDhg3Izc1F79698dZbb2HLli1QqVQ4deoUJEkCAAwdOhTPPvss1qxZA4VCgXPnzsHCwqJMZTI0BiEmwCCEiKhycHZ2Rrdu3bB582ZtELJ9+3a4ubnhxRdfhJmZGYKCgrTp586di127duH//u//MG7cuDKde/PmzcjKysKmTZtga2sLAFi5ciV69OiBhQsXwsLCAqmpqXjllVdQr149AEDjJ9Z/iImJwdSpU9GoUSMAQP28L58KhEGICfj7yz/v35c3FxfTloeIyBRszMyQ3rZtken+SElB94sXi0y3NzAQ7ZycinXekhg6dCjeeustrF69GkqlEt9//z0GDRoEMzMzpKenY86cOdizZw9iY2ORm5uLhw8fIiYmpkTn0CcyMhJBQUHaAAQA2rRpA41GgytXrqBdu3YYMWIEunTpgk6dOiE0NBQDBgyAt7c3APkGsW+++Sa+/fZbhIaG4tVXX9UGKxUFx4SYgK0t4OMjP46KMm1ZiIhMRZIk2CoURW6dXVxQU6mEVFA+AHyVSnR2cSlWfnndFcXVo0cPCCGwZ88e3Lp1C0eOHMHQoUMBAFOmTMGuXbswb948HDlyBOfOnUNgYCBUKlXZ3pxiWr9+PY4fP47WrVtj69ataNCgAU6cOAEAmDNnDi5fvoyXX34Zv//+O5o0aYJdu3aVS7mKi0GIieS1hrBLhoiocApJwrJHfzSfDh/yni/194eihMFFcVlZWaFv3774/vvvsWXLFjRs2BDNmjUDABw9ehQjRoxAnz59EBgYCC8vL9y4ccMg523cuDHOnz+PjIwM7b6jR4/CzMwMDRs21O579tlnMX36dBw7dgzPPPMMNm/erH2tQYMGeO+99/Dbb7+hb9++WL9+vUHKZigMQkxk8WLgzBmgd29Tl4SIqOLr6+6O7QEBqKFU6uyvqVRie0CA0dcJGTp0KPbs2YN169ZpW0EAeZzFzp07ce7cOZw/fx5DhgzJN5OmLOe0srLC8OHDcenSJRw6dAjjx4/H66+/Dk9PT0RHR2P69Ok4fvw4bt68id9++w3Xrl1D48aN8fDhQ4wbNw7h4eG4efMmjh49itOnT+uMGakIOCbERLhIGRFRyfR1d0cvNzccSUlBrEoFb0tLtHVyMloLyJNeeukluLi44MqVKxgyZIh2/5IlSzBq1Ci0bt0abm5umDZtGtLS0gxyThsbG/z666+YMGECWrZsCRsbG/Tr1w9LlizRvv7PP/9g48aNSEpKgre3N8aOHYv//Oc/yM3NRVJSEoYNG4b4+Hi4ubmhb9+++Pjjjw1SNkORREkmTFcTaWlpcHR0RGpqKhwcHExdHCKiSi8rKwvR0dGoU6cOrKysTF0cKqPCrmdJvkPZHWMi6enAF18A770HMAwkIqLqiEGIiZiZAZMmAUuXAklJpi4NEREZ2/fffw87Ozu9W0BAgKmLZxIcE2IiNjZAzZrA7dvyDBk3N1OXiIiIjKlnz54ICQnR+1pFW8m0vDAIMaH69R8HIa1ambo0RERkTPb29rC3tzd1MSoUdseYUN4KulywjIiqC86FqBoMdR0ZhJgQFywjouoir7shMzPTxCUhQ8i7jmXtRmJ3jAnxRnZEVF0oFAo4OTkhISEBgLzGRUmXTyfTE0IgMzMTCQkJcHJygqKYdyMuCIMQE3oyCBEC4OeRiKoyLy8vANAGIlR5OTk5aa9nWTAIMaH69YGTJ+WfDECIqKqTJAne3t7w8PBATk6OqYtDpWRhYVHmFpA8DEJMyNISeO45U5eCiKh8KRQKg32JUeXGgalERERkEgxCTOzQIWD8eGDTJlOXhIiIqHwxCDGxs2eBlSuBPXtMXRIiIqLyxSDExDhNl4iIqisGISb25IJlXEiQiIiqEwYhJla3rnxH3fR0ID7e1KUhIiIqPwxCTEypBGrVkh+zS4aIiKoTBiEVAMeFEBFRdcQgpALIC0Ju3jRtOYiIiMqTJHhf5XzS0tLg6OiI1NRUODg4GP188fGAuTng6mr0UxERERlVSb5DuWx7BeDpaeoSEBERlT92xxAREZFJMAipAIQApkwBXnmF03SJiKj6YHdMBSBJwM6dQHQ0cOUKu2eIiKh6YEtIBcFpukREVN0wCKkgGIQQEVF1wyCkgmAQQkRE1Q2DkAoiLwiJijJtOYiIiMoLg5AK4skghMvHERFRdVAhgpBVq1bBz88PVlZWCAkJwalTpwpM+/XXX6Nt27ZwdnaGs7MzQkND86UfMWIEJEnS2bp27WrsapSJnx+gUMgrp967Z+rSEBERGZ/Jg5CtW7di0qRJmD17NiIiIhAUFIQuXbogISFBb/rw8HAMHjwYhw4dwvHjx+Hr64vOnTvjzp07Oum6du2K2NhY7bZly5byqE6pWVjIa4SkpADu7qYuDRERkfGZ/N4xISEhaNmyJVauXAkA0Gg08PX1xfjx4/HBBx8UebxarYazszNWrlyJYcOGAZBbQlJSUrB79+5Slam87x1DRERUVZTkO9SkLSEqlQpnzpxBaGiodp+ZmRlCQ0Nx/PjxYuWRmZmJnJwcuLi46OwPDw+Hh4cHGjZsiDFjxiApKanAPLKzs5GWlqazERERkXGZNAi5d+8e1Go1PJ9aItTT0xNxcXHFymPatGnw8fHRCWS6du2KTZs2ISwsDAsXLsThw4fRrVs3qNVqvXnMnz8fjo6O2s3X17f0lSqDo0eBXr2AiRNNcnoiIqJyVamXbV+wYAF++OEHhIeHw8rKSrt/0KBB2seBgYFo2rQp6tWrh/DwcHTs2DFfPtOnT8ekSZO0z9PS0kwSiDx4APzf/wEBAeV+aiIionJn0pYQNzc3KBQKxD9117b4+Hh4eXkVeuzixYuxYMEC/Pbbb2jatGmhaevWrQs3NzdEFbAIh1KphIODg85mCnnTdK9fBzQakxSBiIio3Jg0CLG0tETz5s0RFham3afRaBAWFoZWrVoVeNyiRYswd+5c7N+/Hy1atCjyPLdv30ZSUhK8vb0NUm5jqV1bnqKblQU8NdmHiIioyjH5FN1Jkybh66+/xsaNGxEZGYkxY8YgIyMDI0eOBAAMGzYM06dP16ZfuHAhZs6ciXXr1sHPzw9xcXGIi4tDeno6ACA9PR1Tp07FiRMncOPGDYSFhaFXr17w9/dHly5dTFLH4jI3B+rUkR9z+XYiIqrqTD4mZODAgUhMTMSsWbMQFxeH4OBg7N+/XztYNSYmBmZmj2OlNWvWQKVSoX///jr5zJ49G3PmzIFCocCFCxewceNGpKSkwMfHB507d8bcuXOhVCrLtW6lUb++HIBcuwa89JKpS0NERGQ8Jl8npCIy5TohEycCy5YBkycDixeX66mJiIjKrNKsE0L51a8P2NoCubmmLgkREZFxsSVED1O2hOTkyGNDJKlcT0tERGQQJfkONfmYENJlYWHqEhAREZUPdscQERGRSTAIqYAmTAACA4HDh01dEiIiIuNhEFIB/fsvcOkSEBlp6pIQEREZD4OQCihv+XYuWEZERFUZg5AKiEEIERFVBwxCKiB/f/kngxAiIqrKGIRUQHktIf/+C6jVpi0LERGRsTAIqYB8fQFLS0ClAm7dMnVpiIiIjIOLlVVACoU8RVelAlJTTV0aIiIi42AQUkGdPs2l24mIqGpjd0wFxQCEiIiqOgYhREREZBIMQiqoS5eA4GCgeXNTl4SIiMg4OCakgnJwAM6fB8zNgdxc+ScREVFVwpaQCqpmTcDKSg5Abt40dWmIiIgMj0FIBWVmBtSrJz/myqlERFQVMQipwPJWTo2KMm05iIiIjIFBSAXGe8gQEVFVxiCkAuPddImIqCpjEFKBNW4sb3XqmLokREREhseJnxVY27bA33+buhRERETGwZYQIiIiMgkGIZWAEPJ6IURERFUJg5AKbupUwMkJWLXK1CUhIiIyLAYhFZxCAaSlca0QIiKqehiEVHCcpktERFUVg5AKjguWERFRVcUgpILLawm5cQNQqUxaFCIiIoNiEFLBeXsDtraARgNER5u6NERERIbDIKSCkyR2yRARUdXEFVMrgRdfBGrWBOztTV0SIiIiw2EQUgl88YWpS0BERGR47I4hIiIik2AQUkkIAdy/b+pSEBERGQ6DkEogKQlwdgbc3YHsbFOXhoiIyDAYhFQCLi6AWs1pukREVLUwCKkEOE2XiIiqIgYhlQTvIUNERFUNg5BKgkEIERFVNQxCKom8ICQqyrTlICIiMhQGIZUEx4QQEVFVwxVTK4mGDYGuXYFGjeQ1QyTJ1CUiIiIqmwrRErJq1Sr4+fnBysoKISEhOHXqVIFpv/76a7Rt2xbOzs5wdnZGaGhovvRCCMyaNQve3t6wtrZGaGgorlXyJgR3d2DfPnkJdwYgRERUFZg8CNm6dSsmTZqE2bNnIyIiAkFBQejSpQsSEhL0pg8PD8fgwYNx6NAhHD9+HL6+vujcuTPu3LmjTbNo0SIsX74ca9euxcmTJ2Fra4suXbogKyurvKpFRERERZCEEMKUBQgJCUHLli2xcuVKAIBGo4Gvry/Gjx+PDz74oMjj1Wo1nJ2dsXLlSgwbNgxCCPj4+GDy5MmYMmUKACA1NRWenp7YsGEDBg0aVGSeaWlpcHR0RGpqKhwcHMpWQQNLTgZUKsDT09QlISIiyq8k36EmbQlRqVQ4c+YMQkNDtfvMzMwQGhqK48ePFyuPzMxM5OTkwMXFBQAQHR2NuLg4nTwdHR0REhJSYJ7Z2dlIS0vT2SqiTz+VV0+dNcvUJSEiIio7kwYh9+7dg1qthudT/9Z7enoiLi6uWHlMmzYNPj4+2qAj77iS5Dl//nw4OjpqN19f35JWpVzkFauSD28hIiICUAHGhJTFggUL8MMPP2DXrl2wsrIqdT7Tp09Hamqqdrt165YBS2k4XLCMiIiqEpMGIW5ublAoFIiPj9fZHx8fDy8vr0KPXbx4MRYsWIDffvsNTZs21e7PO64keSqVSjg4OOhsFVFeEHL7NvDwoWnLQkREVFYmDUIsLS3RvHlzhIWFafdpNBqEhYWhVatWBR63aNEizJ07F/v370eLFi10XqtTpw68vLx08kxLS8PJkycLzbMycHUFnJzkx9evm7QoREREZWby7phJkybh66+/xsaNGxEZGYkxY8YgIyMDI0eOBAAMGzYM06dP16ZfuHAhZs6ciXXr1sHPzw9xcXGIi4tDeno6AECSJEycOBH//e9/8X//93+4ePEihg0bBh8fH/Tu3dsUVTQY3k2XiIiqEpOvmDpw4EAkJiZi1qxZiIuLQ3BwMPbv368dWBoTEwMzs8ex0po1a6BSqdC/f3+dfGbPno05c+YAAN5//31kZGRg9OjRSElJwQsvvID9+/eXadxIRVG/PvDXXwxCiIio8jP5OiEVUUVeJ+S774ATJ4A+fYCOHU1dGiIiIl0l+Q5lEKJHRQ5CiIiIKrJKs1gZERERVV8MQiqhtDTgzBkgO9vUJSEiIiq9UgUhGzduxJ49e7TP33//fTg5OaF169a4efOmwQpH+tWpA7RoAfzzj6lLQkREVHqlCkLmzZsHa2trAMDx48exatUqLFq0CG5ubnjvvfcMWkDKL2/Rsqgo05aDiIioLEo1RffWrVvwf7Rgxe7du9GvXz+MHj0abdq0QYcOHQxZPtLD3x84eZLTdImIqHIrVUuInZ0dkpKSAAC//fYbOnXqBACwsrLCQ64nbnS8hwwREVUFpWoJ6dSpE9588008++yzuHr1Krp37w4AuHz5Mvz8/AxZPtKDQQgREVUFpWoJWbVqFVq1aoXExETs2LEDrq6uAIAzZ85g8ODBBi0g5ccghIiIqgIuVqZHRV+sLCUFcHaWH6elAfb2Ji0OERGRltEXK9u/fz/+/PNP7fNVq1YhODgYQ4YMQXJycmmypBJwcgLeew9YvBhgCElERJVVqYKQqVOnIi0tDQBw8eJFTJ48Gd27d0d0dDQmTZpk0AKSfkuWAJMnAxWwoYaIiKhYSjUwNTo6Gk2aNAEA7NixA6+88grmzZuHiIgI7SBVIiIiosKUqiXE0tISmZmZAICDBw+ic+fOAAAXFxdtCwkZ18OHwLlzwF9/mbokREREpVOqlpAXXngBkyZNQps2bXDq1Cls3boVAHD16lXUrFnToAUk/XbvBoYMAV54AThyxNSlISIiKrlStYSsXLkS5ubm2L59O9asWYMaNWoAAPbt24euXbsatICkH6fpEhFRZccpunpU9Cm6AJCaKs+SyXtcQYtJRETVTEm+Q0vVHQMAarUau3fvRmRkJAAgICAAPXv2hEKhKG2WVAKOjoC7O5CYKN/IrlkzU5eIiIioZEoVhERFRaF79+64c+cOGjZsCACYP38+fH19sWfPHtSrV8+ghST96tdnEEJERJVXqcaEvPvuu6hXrx5u3bqFiIgIREREICYmBnXq1MG7775r6DJSATguhIiIKrNStYQcPnwYJ06cgIuLi3afq6srFixYgDZt2hiscFQ4f3/5J4MQIiKqjEoVhCiVSjx48CDf/vT0dFhaWpa5UFQ8XboAlpbA88+buiREREQlV6rumFdeeQWjR4/GyZMnIYSAEAInTpzA22+/jZ49exq6jFSAli2B998H2rUzdUmIiIhKrlRByPLly1GvXj20atUKVlZWsLKyQuvWreHv74+lS5cauIhERERUFZWqO8bJyQk//fQToqKitFN0GzduDP+8QQpUbv75B/j7b7lLxsfH1KUhIiIqvmIHIUXdHffQoUPax0uWLCl9iahE3noL+PNPYPNmYPBgU5eGiIio+IodhJw9e7ZY6SRJKnVhqOTq15eDEM6QISKiyqbYQciTLR1UceStFRIVZdpyEBERlVSpBqZSxcG1QoiIqLJiEFLJcdVUIiKqrBiEVHJ5LSFJSUBysmnLQkREVBIMQsqBWgiEJydjS3w8wpOToRbCYHnb2QHe3vJjtoYQEVFlUqp1Qqj4diYmYkJUFG5nZ2v31VQqsczfH33d3Q1yjkWLACurx60iRERElYEkhAH/La8i0tLS4OjoiNTUVDg4OJQ6n52Jieh/+TKefoPzJjFvDwgwWCBCRERUEZTkO5TdMUaiFgIToqLyBSAAtPsmRkUZtGuGiIioMmEQYiRHUlJ0umCeJgDcys7GkZSUMp8rNRX46Sfgu+/KnBUREVG54ZgQI4lVqQyarjAxMUDv3oCzM/Daa2XOjoiIqFywJcRIvC0tDZquMPXqyT+Tk4H798ucHRERUblgEGIkbZ2cUFOpREF30pEA+CqVaOvkVOZz2dgANWrIjzlNl4iIKgsGIUaikCQsezRnVl8gIgAs9feHwkA3/OPKqUREVNkwCDGivu7u2B4QgBpKZb7XWtjZGXR6bl6XzK5dQHg4oFYbLGsiIiKj4DohehhqnZA8aiFwJCUFsSoVcoTA8H/+AQCca9ECQXZ2Zc5/505g1Ch5lkyemjWBZcuAvn3LnD0REVGxcZ2QCkYhSejg7IzBnp4Y5uWFgY9aQObdvFnmvHfuBPr31w1AAODOHXn/zp1lPgUREZFRMAgxgY9q1wYA/JiYiH8yMkqdj1oNTJgA6GvLyts3cSK7ZoiIqGJiEGICgXZ26O3mBgFgfkxMqfM5cgS4fbvg14UAbt2S0xEREVU0DEJM5KNatQAA38fH49+HD0uVR2ysYdMRERGVJ5MHIatWrYKfnx+srKwQEhKCU6dOFZj28uXL6NevH/z8/CBJEpYuXZovzZw5cyBJks7WqFEjI9agdFo4OKCriwvUABaWsjXE29uw6YiIiMqTSYOQrVu3YtKkSZg9ezYiIiIQFBSELl26ICEhQW/6zMxM1K1bFwsWLICXl1eB+QYEBCA2Nla7/fnnn8aqQpnMeDQ2ZH1cHG5nZZX4+LZt5VkwhS014usrpyMiIqpoTBqELFmyBG+99RZGjhyJJk2aYO3atbCxscG6dev0pm/ZsiU+++wzDBo0CEo9a2/kMTc3h5eXl3Zzc3MzVhXKpI2jIzo4OSFHCHx261aJj1co5Gm4QMGByMKFcjoiIqKKxmRBiEqlwpkzZxAaGvq4MGZmCA0NxfHjx8uU97Vr1+Dj44O6deti6NChiCmiuyM7OxtpaWk6W3nJaw35KjYW8aW4mV3fvsD27Y+Xbc+TF3hERJS1hERERMZhsiDk3r17UKvV8PT01Nnv6emJuLi4UucbEhKCDRs2YP/+/VizZg2io6PRtm1bPHjwoMBj5s+fD0dHR+3m6+tb6vOX1EtOTnjewQFZGg2WlKI1BJADkRs3gEOHgM2b5Z9564MsWQKcPGm48hIRERmKyQemGlq3bt3w6quvomnTpujSpQv27t2LlJQUbNu2rcBjpk+fjtTUVO12q5TBQGlIkqRtDVl99y6ScnJKlY9CAXToAAweLP/s2RN47TVAo5FXU83ONlyZiYiIDMFkQYibmxsUCgXi4+N19sfHxxc66LSknJyc0KBBA0RFRRWYRqlUwsHBQWcrT91dXBBsZ4d0tRrLC1v4o4SWLgU8PICgIKAU416JiIiMymRBiKWlJZo3b46wsDDtPo1Gg7CwMLRq1cpg50lPT8f169fhXYHnqT7ZGrL8zh2k5uYaJF9XV+DCBbmLxtHRIFkSEREZjEm7YyZNmoSvv/4aGzduRGRkJMaMGYOMjAyMHDkSADBs2DBMnz5dm16lUuHcuXM4d+4cVCoV7ty5g3Pnzum0ckyZMgWHDx/GjRs3cOzYMfTp0wcKhQKDBw8u9/qVRB83NzS2sUFKbi5W37ljsHyfHHIjhNw9Q0REVBGYm/LkAwcORGJiImbNmoW4uDgEBwdj//792sGqMTExMDN7HCfdvXsXzz77rPb54sWLsXjxYrRv3x7h4eEAgNu3b2Pw4MFISkqCu7s7XnjhBZw4cQLuj24aV1GZSRI+ql0br0VGYsnt23i3Zk3YGnBubVwcMHYs8OyzwIwZBsuWiIio1CQh9N3+rHoryW2IDSlXo0GjU6dwPSsLS+rVw3sGnKWzZQswZAhgYQGcPQsEBBgsayIiIq2SfIdWudkxlZm5mRmmPxob8tmtW8gy4O1vBw0CevQAcnKAkSMBAw07ISIiKjUGIRXM656e8FUqEatSYX0Z1kt5miQBa9bIA1RPn5ZnzhAREZkSg5AKxtLMDNMe3WF3QUwMcgw4krRGDXnxMgCYORO4ds1gWRMREZUYg5AKaJSXF7wsLRGTnY3vnlpHpaxGjgQ6dZLXDXnjDc6WISIi02EQUgFZKxSY8mhQ6ryYGKgNOHZYkoCvvwbs7ICYGMCAa6MRERGVCIOQCuo/3t5wNTdH1MOH2JaQYNC8a9cG9uwBLl0CHvX8EBERlTsGIRWUnbm5dorupzdvQmPgmdTt2smtIURERKbCIKQCG1ejBhwVClzOzMRP9+4Z5RwajTxr5ttvjZI9ERFRgRiEVGCO5uYYX7MmAOC/N2/CGOvKbd0KvPMOMG4cYMDV4omIiIrEIKSCm1CjBmzNzBCRno799+8bPP8BA4CQECAtDXj7bfn+MkREROWBQUgF52ZpiTE1agAA5hqhNUShANatAywtgV9+ke+4S0REVB4YhFQCk2vWhFKScDwtDeEpKQbPv0kTYPZs+fG77wIGXpqEiIhILwYhlYCXUom3fHwAyGNDjGHqVCA4GLh/Xx4fQkREZGwMQiqJqb6+sJAk/J6SgmOpqQbP38ICWL8eMDcHdu6U1xAhIiIyJgYhlUQtKysM9/ICIK8bYgzBwcCKFcDJk8AzzxjlFERERFoMQiqRD2rVghmAvffvI+LBA6Oc4+23gRYtjJI1ERGRDgYhlUg9a2sM8fQEYLzWkCddvgwcPmz00xARUTXFIKSSmV6rFiQAO+/dw+WMDKOdJywMePZZYMgQwAgTcoiIiBiEVDZNbG3Rz90dADDPiK0hrVsDfn7A3bvAlClGOw0REVVjDEIqoY8e3fr2h4QEXMvMNMo5rK2Bb76RH3/zDXDggFFOQ0RE1RiDkEoo2N4er7i6QgNgQUyM0c7Ttu3jNUPeekvulgkPB7ZskX+q1UY7NRERVQOSMMZd0Sq5tLQ0ODo6IjU1FQ4ODqYujl4nUlPR6uxZmEsSokJCUNvKyijnSU+Xp+vevAnY2gJPDkOpWRNYtgzo29copyYiokqoJN+hbAmppJ53dESoszNyhcAiI7aG2NkBw4fLj58eB3vnDtC/v7y4GRERUUkxCKnEZtSuDQD4JjYWd7OzjXIOtVq+wZ0+eW1oEyeya4aIiEqOQUgl1s7RES84OiJbCHx+65ZRznHkCHD7dsGvCwHcuiWnIyIiKgkGIZWYJEna1pA1d+5gd2IitsTHIzw5GWoDDfWJjTVsOiIiojzmpi4AlU1nZ2fUs7LC9aws9Ll8Wbu/plKJZf7+6PtoTZHS8vY2bDoiIqI8bAmp5Hbdu4frWVn59t/Jzkb/y5exMzGxTPm3bSvPgpEk/a9LEuDrK6cjIiIqCQYhlZhaCEyIitL7Wl5nzMSoqDJ1zSgU8jRcoOBAZOlSOR0REVFJMAipxI6kpOB2IbNiBIBb2dk4Usabv/TtC2zfDtSokf+1ESO4TggREZUOg5BKLFalMmi6wvTtC9y4ARw6BGzeDEyeLO/ftQu4d6/M2RMRUTXEIKQS87a0NGi6oigUQIcOwODBwMKFQFCQvJT77NkGyZ6IiKoZBiGVWFsnJ9RUKlHAUA0AgI+lJdo6ORn83AqFPBYEAK5f52JlRERUcgxCKjGFJGGZvz8AFBiICACJBuiO0adDB+DECWDfPg5MJSKikmMQUsn1dXfH9oAA1FAqdfZ7W1rCzdwcsSoVOp4/b7RAJCSk4FkzREREheFddPWoDHfRfZpaCBxJSUGsSgXvR10wN7Ky0O7sWdxVqRBka4vfg4PhYmFhlPMnJQHLlwMffQQYaAgKERFVQiX5DuWKqVWEQpLQwdlZZ189a2v8HhyM9mfP4nxGBrpcuICDQUFwNDfsZddo5MXKIiMBR0dg0iSDZk9ERFUUu2OquIY2NggLDoabhQX+evAA3S5cwIPcXIOew8zs8ZTdTz4ByrhIKxERVRMMQqqBAFtbHGjaFM7m5jieloZXLl5EhoGns4wYATz7LJCaCsyaZdCsiYioimIQUk0E29vjt6ZN4aBQ4I/UVPS6eBEPDRiIKBTAF1/Ij7/6Crh40WBZExFRFcUgpBpp4eCA/U2bwtbMDGEpKeh3+TKyNRqD5d++PdCvnzxG5L33AA55JiKiwjAIqWZaOTpib9OmsDYzw7779zHw8mXkGDAQ+ewzeXZMWBjw888Gy5aIiKogBiHVUDsnJ/zfM89AKUn4KSkJQyMjkWugQKROHWDqVOA//wFatTJIlkREVEVxnRA9KuM6IaWxNykJvS9dQo4QGOrhgY2NG0NhgJXHhOACZkRE1VVJvkPZElKNdXd1xY8BATCXJHyfkIDRV65AY4CY9OkAJCenzFkSEVEVZPIgZNWqVfDz84OVlRVCQkJw6tSpAtNevnwZ/fr1g5+fHyRJwtK8O6iVIc/qrpebGzY3bgwzAOvi4jDu2jUYqnHs6lWge3dg/HiDZEdERFWMSYOQrVu3YtKkSZg9ezYiIiIQFBSELl26ICEhQW/6zMxM1K1bFwsWLICXl5dB8iTgVQ8PbGrcGBKANXfv4r2oKIMEIvHx8s3tvv4auHCh7OUkIqKqxaRjQkJCQtCyZUusXLkSAKDRaODr64vx48fjgw8+KPRYPz8/TJw4ERMnTjRYnnmqy5iQp62LjcUbV64AAKb5+mJ+3bqQyji4Y8AA4McfgZdeAg4e5FgRIqKqrlKMCVGpVDhz5gxCQ0MfF8bMDKGhoTh+/Hi55pmdnY20tDSdrToa5e2NNfXrAwAW3rqFOTdulDnPRYsApRL4/Xfgp5/KnB0REVUhJgtC7t27B7VaDU9PT539np6eiIuLK9c858+fD0dHR+3m6+tbqvNXBW/XqIGl/v4AgE9u3sS8mzfLlJ+f3+P7ykyZAmRnl7GARERUZZh8YGpFMH36dKSmpmq3W7dumbpIJjWhZk0srFsXAPBRdDSWlPH9mD4d8PYGrl8Hli83RAmJiKgqMFkQ4ubmBoVCgfj4eJ398fHxBQ46NVaeSqUSDg4OOlt1936tWvjYzw8AMPn6day6cwdqIRCenIwt8fEIT06GupjDiezsgPnz5cc//CAv605ERGSyIMTS0hLNmzdHWFiYdp9Go0FYWBhalXKpTWPkWZ3NrF0bH9aqBQAYd+0a3I8exYvnz2NIZCRePH8efidOYGdiYrHyev11YN064NgxwIztb0REBBN3x0yaNAlff/01Nm7ciMjISIwZMwYZGRkYOXIkAGDYsGGYPn26Nr1KpcK5c+dw7tw5qFQq3LlzB+fOnUNUVFSx86TikyQJ/61TBz1dXQEAybm5Oq/fyc5G/8uXixWImJkBI0fKg1SJiIgAwNyUJx84cCASExMxa9YsxMXFITg4GPv379cOLI2JiYHZE/823717F88++6z2+eLFi7F48WK0b98e4eHhxcqTSkYDIOLBA72vCQASgIlRUejl5lbsJd9zcoDdu4H+/Tlll4ioOuO9Y/SoruuE6BOenIwXz58vMt2hoCB0cHYuMp1aDTz3HBARAezcCfTpY4hSEhFRRVEp1gmhyiFWpSpWusuZmcVKp1DIS7kDnLJLRFTdMQihQnlbWhYr3bvXrmFYZCTOp6cXmXbaNHnK7r//AsuWlbWERERUWTEIoUK1dXJCTaUShQ3dUEoSNAC+jY9H8F9/IfTcOexNSirwjrx2dsCCBfLj//5XvscMERFVPwxCqFAKScKyRyuoPh2ISI+2zU2a4HSzZhjk4QEFgLCUFLx88SKeOX0a/7t7F1lqdb58X3sNaNkSePAAmDHD2LUgIqKKiANT9eDA1Px2JiZiQlQUbj8xiMNXqcRSf3/0dXfX7ovJysLy27fxVWwsHjwKPjwsLDC2Rg2M8fGB+xPdO8eOAW3ayDNkIiKA4OByqw4RERlJSb5DGYTowSBEP7UQOJKSgliVCt6Wlmjr5FTgtNy03Fz8LzYWy27fRsyjwMXKzAzDPT3xnq8vGtrYAACGDAFiY4HVq4HGjcutKkREZCQMQsqIQYjh5Go02J6YiM9v38ZfT6w38oqrKybXrImWlk6wsZG064WUJNAhIqKKh0FIGTEIMTwhBI6kpmLJrVv4v6Qk5P3SNbOzw2RfX7zq7o6fk5LydfnUVCqx7KkuHyIiqrgYhJQRgxDjupqZiaW3b2NDXBwePrqbnbPC/PGy8E80fOQ93B4QwECEiKgS4GJlVKE1sLHB6gYNEPP885jr5wcPc3Mkq3MfT7d5ggAghLw0fHHv2ktERJUDgxAyGTdLS8zw88O3TZoUnlACbmVn40hKSrmUi4iIyodJb2BHBACJ2TnFSvdTYhJaOzrC0qxssbNaDRw5Is/K8fYG2raVl5MnIqLyxSCETC7xSvGWhl969za+iolDO7UHBjl7onsdB7i5SSW6E+/OncCECcDt24/31awpLx/ft28JC05ERGXC7hgyOfdYJyBBCWgKSKABkKGAvcoCmea52K+8ixGZZ+Gx/xQs37yBOq0fomNH4PvvHx+SnQ38/be8ImuenTuB/v11AxAAuHNH3r9zp4ErRkREhWIQQiZXw1sCVvrLg1KfDkQ0kPcvbIQv7rRGm11N4XrGE8gyA3wfIvf1G7gx7yR+73UWu1R3kZIjd+38/TcQEAA4OABOTkBgIDB0qDzI9Wl5+yZOlLtqiIiofHCKrh6colu+1GrAzw+4XTcRGBsFeDxeJwTxSmC1P3yj3REd/XjsRnpuLrbF3cO6W/E4lp2sXXdEKUno6eaGoARPfPaqC1KT9MTZZgIITAFcVUCSJXDRCdDIfTqHDgEdOpSxPuWw4BrHtRBRRcV1QsqIQUj5y+sqEWYCeCblcYBwyQmSRsL27QWP2biTnY3N8fHYFB+PSxkZ2v1uFhbo5+SBF3M9EbHFHosWSkDbRGDcU4FOglJuiTnijs2bgcGDy1APPffYMfSCaxzXQkQVGYOQMmIQYhr6vlx9fYGlS4v35SqEwPn0dHwbH4/v4+MRn/N41k0t2CDmoB3QMUHe8WTDRF6Xz+wAdLNzx7JlQP36pSh/YiL6X76Mpz9QhlxwTRusPXWSvIaWwoI1IqLywCCkjBiEmI6huhlyNRocTE7Gpvh47L53T7sya4EEgPsWwOQgSGoz9OxuhnfHSnimkQQLSYK5JP+0MDPT27WiFgJ+J07otIA8SYLcIhL9/POl7prRdlvd1v+6JMktIk92WxERlTcGIWXEIKRqScvNxbybN7Hw1i2D5CcBj4OSR4GJWojHy84X4lBQEDo4O5fqvOHhwIsvFp3OEONaqGQ4RofosZJ8h3KdEKryHMzNEWRnV6y0tmZmgEbCw1wBjaQB9HyRCAA5QiAnL34vwZSaEf/8g04uLmhpb48W9vZ4xta2WIuvZWUB+/Y9saOQwbWxscUuTqF4R+Pi4RgdotJjEELVgrdl8RZE+yUwUNtSERkJ+PsLQCEHHCvXCvyyT2DCZA1aPi+QC4HcR8HI8dRUvHX1apH538zOxv9iY/G/R5GCpSQhyM4OLR4FJS3s7dHExgbmTwUmL7wAnDnz6EkRg2szM4tV1UKVxwDbqqCgMTp5a89wjA5R4dgdowe7Y6qevDEbd7Kz8w0cBYoes6HRyINV//1Xft6qFTBjBtCtmzwWozj5e1laYrm/P86mp+P0gwf468EDvV045mozWN+2w2vP2eN5ZzkwWf+JDbZ8L+Fe40Rkf3j5cabaAsrP7T4LQPo+d/TtC3z8MfDMMyV7n4DyGWBbFVTZMTrnzgHTpwPz5wPBwaYuDVVCHBNSRgxCqqa8L1cAOl+wxf1yjYkBFi0C/vc/eUVWAHj2WTkY6d0b2J1UsvyFELiYlIVNJx7gt5sP8I94gBy/B4Bd/u4dOzMFnrWzxemUDGRJ6nx3GwYAaADrDCUe9nkeUMvL2Q8cCMyaJeDfUCBbo0G2ePQzb3vq+UONBiP/+QdJBYxvMcQA2yepcgVW/5mC68kq1HO2xDsvOMHS3MBrqhipW6nKjtGZORP473/ln598YurSUCXEIKSMGIRUXfq6GXyVSiwtQTdDbCywZAmwZg2QtyzJuHHAihXA+z8nYokqCmrXx/krkpSYZOmPRT1089+6FRg2DFCpHu+zcxBo8+pD1O76AFLDB7ic+wARDx4gs6jZPU9wliyQmQFkazSApQYwFwZfGznAxgZBdnbws7LSbrWtrFBLqYRVMf/tL8l7VVrG6lZKTwe++AKYNavotJs2Aa+/XupTlb/gYOD8efnn2bOmLg1VQgxCyohBSNVmqP+M792TBx+uWAEcOADcuvVofID01KDRS06AWkLv3sCbbwIvvywfn7e0fL16QI8ewCuvyLMqnh6+ohYC/2RmYuWdO1h7925Zqw8zAEozM3mTpMePzcyQkZuL6AKmGReXt6VlvuAk73EtpRLWCgXe/zkRn9kV3K00NT2gzIGIobuVcnKA334DNm8Gdu+Wu1ueWBuvQA4OwJw5wHvvFftUepXLQOH4eMDLS/e5h4dhz2FknKlkegxCyohBCJVERgZgZVX4+IA8r74KbNsmPxYCiIoC/P1RrDsBhycn48Xz54tMt7Z+fbR2dNQGFpEXJCxZYIbJ75ohtL28zklKivzffM2apTvH7Nq1YatQ4GZWFm48sWUUo8XG09wC8Q9z5RYaffUWgFmqBSJeCoKDhQJWj+qR97M4X7yGWrdFCODYMTnw2LZNDjzz1K0LZGbK39P5As9Hs5XMzOTxRCtXAmPHysclJwNXrgAhIcW77kA5DhTetAkYPlz3uQGbcYwdSO3cCbw7UeCOS4r2WtS474TlSyWDDhAul1szVOJzcIouUTmytZXHBxQVgABA06aPH0tSyVZmbevkhJpKZZGDa9/08dH5Q1L7OaDrU3cIXrRI7lL6z3/kMYh5//wW9xwz/fzy/bESQiApJwc3s7N1ApMbWVm4mZWF6KwspKvViM/NASwKqagEaJxyEBzxl96XLSRJJzB5clM+ei1DrS4wAAHkMTu3srOxKzERvdzcYFHANOnPPgOmTXv83MMDGDRIvhliy5bArl1Av2V67nmUoARW+eOHce5wdtYd37ltG/D220Dt2nJeAwfKrxf0t7+gFp072dnof/myQQcKa/bsARQKmKnV0CgUwJ49MDNQEGLsQGrnzkfXYrHutbiToES/Zf7YAXeDBCLlcmuGxERMuBaF26onzmGpxLL6lescxcGWED3YEkIltWULMGRI0ekMcW+asgyuBeT/7l95Bdi7V35ubS3/l/7++4C7u2HO8aSMDHmdExcXeUG3QVvu4kCt6CKPU2rMYG4OPFRr8pZAMRonc3O4SBbQ3LeEr70FmnhbwMPSEpr7Flg8wxLtm1qgb0cL9HjBEp7WFtoAbGdiIvpdKrhbaccz+d+rBQuATz+VW6LyNGjwOCBp0uRRFkIgU61Gw1OncPfJgUNPKPFA4Tt35KYbPX6/fx8te/WC/RNzvNNsbfHX7t14ycVFf36enkCNGkWe1tgzrtRqwLN/IpLeLfhauK4IQPyP7mXqmimXWzOU4neqop2D3TFlxCCESqo8Z0oYYnCtEMDvv8sTII4fl/fZ2gLvvgtMmQIsOFryQaNCAHFx8gzPJ7dr14CJE+WWFwCYvy8ZH1oX3eXzBYIQonRG69YAzDTw9hMICNagUVMN6j+jQd2GGnj6apADDbI0j7dsjQbn0tOxqBgr5JpB/rtbEhIAVwsLuJub49+sLGQX8ifU1swM/dzdkfPkLCQh8DBXg/j7GiSmapCWKSDMNYCFPJDY1llAJTSPF8MrhlBnZwTZ2sLD0hKelpbwtLDQPvawsHjc0tOxo3zhC6CRJJg9cd6nn+fTsSNw8GChZdN2jWVlF9j95mtVdCCl0QB378rT5J/c+vYFHF0EQhNOAO4FnEMDIFGJg57Po2OHUt42oTxuzSAEPH8/gSSzguvhKpSIf0n/OYQQEJADWAHofZwrBOodOY37kqpU5ygOBiFlxCCESipvzYg7d/IvXAUYfs0IQ/XlCgHs3y/P8vjrUe9Hly7yAEx9A2zz7mjcsyeQkgK4ucnHpKTI/80nJuo/T9++wI4d8mNVroD1Tyegcc7WP2tHAyiSlcjs9TzCf5cwdqw8dkafZcvkwAmQz33tGhAYCNjYPfpjLhV8DstUJdRDQqC2VANOKsApB41aqdAiNAf+LXKQoFIhIScHiY9+JqhUBU5druiczc3haWmJAeHhmPrpp7BNS9P7/VNcAoDa0RGRX3yB2z17aoO/bCEeP34UFF7NzMR3CQlF5vmcvT3sYI6MhwJmCsDCSkAtBNIfAleuCmTnCAgJ8mrBikfjicwEHF0EhJUaaYqcok4Bixwz2FuXbqpYjkaDB8UY82RnZqYN+or6cn366zdXiGKNq7IAAEnKF2QY0sHAIHR0Ld0tJhiElBGDECqNvNUzAd1ApDLc4VYI4Oef5eUhbt8ufOl3Cwu5Tp07y8fkHe/pCSQlAY0ayWMcgoIe//T01M1DOztGQDdIKGB2zIMHwMWLcsvK+fPyz4sX5S6lvJalb7+VpzwD8oyjW36JUH1U8DkwOwA44o7AQHmMx6BB8jiNwuRqNEjKzUWCSoUfEhIwLyam8AMADPLwQIi9vc4spKdnJeU9t4AZrM3l50nxZggemAYsuFTkOd708IZSKHBf5CBJrUK8SoX4RwHU06vOuCcnY80XX6DfkSNFt3Q8JS/9jrZtMea995BYyvsgUcU3A40xt4Nn0Qn14MBUIhPo21cONPTdR2Tp0oobgAByUNGzpzydtKhupZxH/3A+2TohSfK0yFq15DEmRVnUwx34OSB/l0+y/i4fe3ugdWt5y/P0LXuysgAfH7nJ/vp1ANfdAVVA/iXuE+VBozjijm++AUaNKrq8eczNzOTuDktLJOXkFCsI+Y+3d6luWmjvBLzbyhXLE5SAW8EtOkhUovnVBhjzn8dtG1ZW8uZuI2DpmoPB7+Sg80AVElQqXEnMwQcvb8Ch+j/hvz/Mhl1WJsw1Rd//KNfMDA9sbDB23FTsfL4TpFwJXg/N4OMuB1JSjhlOHTWDyJagzjIDVGZAjhngkAO0u1dk/tjsC9ywhZ2NhA7tgWFDJSgkCWaQcPkiUMNbgoebBEsF5P2SBAXkxxFp6Rh3/VqRp1jXoCGed5S/FMe8AxwOB+o3kBcbfOVleUxUQU6lpWHElStFnmNjw4Z4zsEBUgEtk0/vffAAOHESOHYU+OV6GtLG/VPkOTCnMXDZERASwsKAJo0AM0nC558DixZI0DaNCN3Hh34H9t9NwULXi0WfI6l4t7ooK7aE6MGWECqLyrxOQXEH2C5ZIgdbxbj3XqGMsWJqYqK8kNj8+Y92FHKzv7IMFC7rrQCKY8sWYMiaRODjwlt05nR0x5w5Beczd668si8AXLggt04BgDsSsMF+KLo9OFho94wAsM8+FCMefI9EPF43JG+RPiD/EiNaZgLYcqLIQOrT6Ocx7h0JpfmTqx1LUUj325PjHDQaudVs+/bHqx+bmcldkcOHA716yUHc0+cw1PW+fl1uufvtN+DUqScC6mK+V7ttn0e7NvI57O0B80fNCZmZ8ibE49bYJx+7uABHjj0aP1PEOcoyfoYtIUQmpFBUsmW6n+DtXbx0zz5b9gAEACzNJUzsYNgmfXd3uatIG4RoJOC8/nMUt776KCQJy/z90f/yZUjQP5Noqb9/mdZd8PYGcMRd7joqpEWn3cfyyrsPH8otQk/+fPhQdwKLjw/w9dd5r3sgc30Icq8cgkUhrSFqMwWy6zyPT8Z4aFtZrKzkbq88rq7yTR+ffN3KCjh2TELHOf5yIKWB/kBqlT9af1K6AASQr8VXTf3lGR8FnOOrwMfXwswM+O47OYDatg3YuFEeoL1vn7zpG29b2ustBHD1qvxe5HX3Xbsm39spT/36QKdOwEsvSXhrkz+SJxZcD9et/njlR0nvPzY2NvJWmA5tJbj295dnEhVyjg4/GnlK2iNsCdGDLSFUXZX3AFtjKc96GGK2UkF06qFnQTRJSGWuR7p/MGyuny90ZX8NgAz/YNhfK/ky7tob/dXVs55KvBJY7Q/faHfDXYun174o5hoeV6/Ka7N9+628Nsw778j7k5PlxeaGDZODiOLcbuDePTmIOXDg8WrKU6fK6/MAcmvFG2/IwU6nTrpjkbTrnRSw9syOCWVf78TY5yjRd6igfFJTUwUAkZqaauqiEJW7HTuEkCR5e9yY+3jfjh2mLmHxlGc9cjUacej+fbE5Lk4cun9f5Go0BsvbqPWIjdXNFBDqRydSP31CQIi4uDLVAQqNQNB9gZfi5J8KTYW7Fmq1ENnZj5+vWfO4+gEBjx6b6a9H795CNGuW/1pZWgoxZkzxy7BjhxA1fHXPUbOWxqDvkzHPUZLvUAYhejAIoepuxw4hatbU/UPq61t5ApA8rEcRNmzQyTQHCpGkcBRz/CaJJIWjyIFC96QbN1a8OhjZvn1CdOyYPx57epMkIczMHj8PDBRi8mQh9u8XIiOj5OfNzRXi0CEhNm+Wf+bmGrpmxjtHSb5D2R2jB7tjiCr3ANsnsR6FGDhQHp356Ltzn1UfDM9ai0R4wB0J2Gj9Nro93CX3X0mSfPOjH36oWHUoJ1u3ytO4i/Lhh8D48QUM0q0mODCViMqsMg+wfRLrUYDcXHmlOo0GcHICvvwSnfsNwDZtkOCBtm13Aju2yTcZSkmRR22q1aWOHCrztSjGGmIAgGeeqd4BSEkxCCEiqo4ePpRvBVynDrB2LeDhAQX0BAkDBsg7334buHFDHlVpb1/uxTW14s6kKsuMq+qI3TF6sDuGiKqFkrZqlKEVpLKrKjPHykNJvkMNMNOfiIgqpZJ+W1bjb1eFQr5XEfD4Vgx58p4vXVqt36JSYRBCRERUDHm3Znhy8TdAbgGpyPeGqsg4JoSIiKiY+vaVl3WvrLN8KpoK0RKyatUq+Pn5wcrKCiEhITh16lSh6X/88Uc0atQIVlZWCAwMxN69e3VeHzFiBCRJ0tm6du1qzCoQEVE1kTfLZ/Bg+ScDkNIzeRCydetWTJo0CbNnz0ZERASCgoLQpUsXJCQk6E1/7NgxDB48GG+88QbOnj2L3r17o3fv3rh0Sfd21127dkVsbKx227JlS3lUh4iIiIrJ5LNjQkJC0LJlS6xcuRIAoNFo4Ovri/Hjx+ODDz7Il37gwIHIyMjAL7/8ot33/PPPIzg4GGvXrgUgt4SkpKRg9+7dpSoTZ8cQERGVTqWZHaNSqXDmzBmEhoZq95mZmSE0NBTHjx/Xe8zx48d10gNAly5d8qUPDw+Hh4cHGjZsiDFjxiApKanAcmRnZyMtLU1nIyIiIuMyaRBy7949qNVqeHp66uz39PREXFyc3mPi4uKKTN+1a1ds2rQJYWFhWLhwIQ4fPoxu3bpBrdZ/q+r58+fD0dFRu/n6+paxZkRERFSUKjk7ZtATC/wHBgaiadOmqFevHsLDw9GxY8d86adPn45JkyZpn6elpTEQISIiMjKTtoS4ublBoVAgPj5eZ398fDy8Clh838vLq0TpAaBu3bpwc3NDVFSU3teVSiUcHBx0NiIiIjIuk7aEWFpaonnz5ggLC0Pv3r0ByANTw8LCMG7cOL3HtGrVCmFhYZg4caJ234EDB9CqVasCz3P79m0kJSXBu5iL+ueN1eXYECIiopLJ++4s1rwXYWI//PCDUCqVYsOGDeLvv/8Wo0ePFk5OTiIuLk4IIcTrr78uPvjgA236o0ePCnNzc7F48WIRGRkpZs+eLSwsLMTFixeFEEI8ePBATJkyRRw/flxER0eLgwcPimbNmon69euLrKysYpXp1q1bAgA3bty4cePGrZTbrVu3ivy+NfmYkIEDByIxMRGzZs1CXFwcgoODsX//fu3g05iYGJiZPe41at26NTZv3owZM2bgww8/RP369bF7924888wzAACFQoELFy5g48aNSElJgY+PDzp37oy5c+dCqVQWq0w+Pj64desW7O3tIT19k4AqJG/sy61bt6pdF1R1rTvrzXpXB6y3aesthMCDBw/g4+NTZFqTrxNCplOd10OprnVnvVnv6oD1rjz1NvmKqURERFQ9MQghIiIik2AQUo0plUrMnj272GNlqpLqWnfWm/WuDljvylNvjgkhIiIik2BLCBEREZkEgxAiIiIyCQYhREREZBIMQoiIiMgkGIRUYfPnz0fLli1hb28PDw8P9O7dG1euXCn0mA0bNkCSJJ3NysqqnEpsGHPmzMlXh0aNGhV6zI8//ohGjRrBysoKgYGB2Lt3bzmV1nD8/Pzy1VuSJIwdO1Zv+sp6rf/44w/06NEDPj4+kCQJu3fv1nldCIFZs2bB29sb1tbWCA0NxbVr14rMd9WqVfDz84OVlRVCQkJw6tQpI9WgdAqrd05ODqZNm4bAwEDY2trCx8cHw4YNw927dwvNszSflfJW1PUeMWJEvjp07dq1yHwr+vUGiq67vs+7JEn47LPPCsyzol1zBiFV2OHDhzF27FicOHECBw4cQE5ODjp37oyMjIxCj3NwcEBsbKx2u3nzZjmV2HACAgJ06vDnn38WmPbYsWMYPHgw3njjDZw9exa9e/dG7969cenSpXIscdmdPn1ap84HDhwAALz66qsFHlMZr3VGRgaCgoKwatUqva8vWrQIy5cvx9q1a3Hy5EnY2tqiS5cuyMrKKjDPrVu3YtKkSZg9ezYiIiIQFBSELl26ICEhwVjVKLHC6p2ZmYmIiAjMnDkTERER2LlzJ65cuYKePXsWmW9JPiumUNT1BoCuXbvq1GHLli2F5lkZrjdQdN2frHNsbCzWrVsHSZLQr1+/QvOtUNe8RHebo0otISFBABCHDx8uMM369euFo6Nj+RXKCGbPni2CgoKKnX7AgAHi5Zdf1tkXEhIi/vOf/xi4ZOVrwoQJol69ekKj0eh9vSpcawBi165d2ucajUZ4eXmJzz77TLsvJSVFKJVKsWXLlgLzee6558TYsWO1z9VqtfDx8RHz5883SrnL6ul663Pq1CkBQNy8ebPANCX9rJiavnoPHz5c9OrVq0T5VLbrLUTxrnmvXr3ESy+9VGiainbN2RJSjaSmpgIAXFxcCk2Xnp6O2rVrw9fXF7169cLly5fLo3gGde3aNfj4+KBu3boYOnQoYmJiCkx7/PhxhIaG6uzr0qULjh8/buxiGo1KpcJ3332HUaNGFXoTxqpwrZ8UHR2NuLg4nevp6OiIkJCQAq+nSqXCmTNndI4xMzNDaGhopf4dSE1NhSRJcHJyKjRdST4rFVV4eDg8PDzQsGFDjBkzBklJSQWmrarXOz4+Hnv27MEbb7xRZNqKdM0ZhFQTGo0GEydORJs2bbR3HNanYcOGWLduHX766Sd899130Gg0aN26NW7fvl2OpS2bkJAQbNiwAfv378eaNWsQHR2Ntm3b4sGDB3rTx8XFae/anMfT0xNxcXHlUVyj2L17N1JSUjBixIgC01SFa/20vGtWkut57949qNXqKvU7kJWVhWnTpmHw4MGF3sispJ+Viqhr167YtGkTwsLCsHDhQhw+fBjdunWDWq3Wm74qXm8A2LhxI+zt7dG3b99C01W0a25ukrNSuRs7diwuXbpUZN9fq1at0KpVK+3z1q1bo3Hjxvjyyy8xd+5cYxfTILp166Z93LRpU4SEhKB27drYtm1bsf5LqAq++eYbdOvWrdBbaVeFa0355eTkYMCAARBCYM2aNYWmrQqflUGDBmkfBwYGomnTpqhXrx7Cw8PRsWNHE5asfK1btw5Dhw4tcnB5RbvmbAmpBsaNG4dffvkFhw4dQs2aNUt0rIWFBZ599llERUUZqXTG5+TkhAYNGhRYBy8vL8THx+vsi4+Ph5eXV3kUz+Bu3ryJgwcP4s033yzRcVXhWudds5JcTzc3NygUiirxO5AXgNy8eRMHDhwo8e3ci/qsVAZ169aFm5tbgXWoStc7z5EjR3DlypUSf+YB019zBiFVmBAC48aNw65du/D777+jTp06Jc5DrVbj4sWL8Pb2NkIJy0d6ejquX79eYB1atWqFsLAwnX0HDhzQaSWoTNavXw8PDw+8/PLLJTquKlzrOnXqwMvLS+d6pqWl4eTJkwVeT0tLSzRv3lznGI1Gg7CwsEr1O5AXgFy7dg0HDx6Eq6trifMo6rNSGdy+fRtJSUkF1qGqXO8nffPNN2jevDmCgoJKfKzJr7mpR8aS8YwZM0Y4OjqK8PBwERsbq90yMzO1aV5//XXxwQcfaJ9//PHH4tdffxXXr18XZ86cEYMGDRJWVlbi8uXLpqhCqUyePFmEh4eL6OhocfToUREaGirc3NxEQkKCECJ/nY8ePSrMzc3F4sWLRWRkpJg9e7awsLAQFy9eNFUVSk2tVotatWqJadOm5XutqlzrBw8eiLNnz4qzZ88KAGLJkiXi7Nmz2lkgCxYsEE5OTuKnn34SFy5cEL169RJ16tQRDx8+1Obx0ksviRUrVmif//DDD0KpVIoNGzaIv//+W4wePVo4OTmJuLi4cq9fQQqrt0qlEj179hQ1a9YU586d0/m8Z2dna/N4ut5FfVYqgsLq/eDBAzFlyhRx/PhxER0dLQ4ePCiaNWsm6tevL7KysrR5VMbrLUTRv+tCCJGamipsbGzEmjVr9OZR0a85g5AqDIDebf369do07du3F8OHD9c+nzhxoqhVq5awtLQUnp6eonv37iIiIqL8C18GAwcOFN7e3sLS0lLUqFFDDBw4UERFRWlff7rOQgixbds20aBBA2FpaSkCAgLEnj17yrnUhvHrr78KAOLKlSv5Xqsq1/rQoUN6f6/z6qbRaMTMmTOFp6enUCqVomPHjvnej9q1a4vZs2fr7FuxYoX2/XjuuefEiRMnyqlGxVNYvaOjowv8vB86dEibx9P1LuqzUhEUVu/MzEzRuXNn4e7uLiwsLETt2rXFW2+9lS+YqIzXW4iif9eFEOLLL78U1tbWIiUlRW8eFf2aS0IIYdSmFiIiIiI9OCaEiIiITIJBCBEREZkEgxAiIiIyCQYhREREZBIMQoiIiMgkGIQQERGRSTAIISIiIpNgEEJEREQmwSCEiKqF8PBwSJKElJQUUxeFiB5hEEJEREQmwSCEiIiITIJBCBGVC41Gg/nz56NOnTqwtrZGUFAQtm/fDuBxV8mePXvQtGlTWFlZ4fnnn8elS5d08tixYwcCAgKgVCrh5+eHzz//XOf17OxsTJs2Db6+vlAqlfD398c333yjk+bMmTNo0aIFbGxs0Lp1a1y5csW4FSeiAjEIIaJyMX/+fGzatAlr167F5cuX8d577+G1117D4cOHtWmmTp2Kzz//HKdPn4a7uzt69OiBnJwcAHLwMGDAAAwaNAgXL17EnDlzMHPmTGzYsEF7/LBhw7BlyxYsX74ckZGR+PLLL2FnZ6dTjo8++giff/45/vrrL5ibm2PUqFHlUn8i0sNk9+8lomojKytL2NjYiGPHjunsf+ONN8TgwYO1tyz/4YcftK8lJSUJa2trsXXrViGEEEOGDBGdOnXSOX7q1KmiSZMmQgghrly5IgCIAwcO6C1D3jkOHjyo3bdnzx4BQDx8+NAg9SSikmFLCBEZXVRUFDIzM9GpUyfY2dlpt02bNuH69evadK1atdI+dnFxQcOGDREZGQkAiIyMRJs2bXTybdOmDa5duwa1Wo1z585BoVCgffv2hZaladOm2sfe3t4AgISEhDLXkYhKztzUBSCiqi89PR0AsGfPHtSoUUPnNaVSqROIlJa1tXWx0llYWGgfS5IEQB6vQkTljy0hRGR0TZo0gVKpRExMDPz9/XU2X19fbboTJ05oHycnJ+Pq1ato3LgxAKBx48Y4evSoTr5Hjx5FgwYNoFAoEBgYCI1GozPGhIgqNraEEJHR2dvbY8qUKXjvvfeg0WjwwgsvIDU1FUePHoWDgwNq164NAPjkk0/g6uoKT09PfPTRR3Bzc0Pv3r0BAJMnT0bLli0xd+5cDBw4EMePH8fKlSuxevVqAICfnx+GDx+OUaNGYfny5QgKCsLNmzeRkJCAAQMGmKrqRFQIBiFEVC7mzp0Ld3d3zJ8/H//++y+cnJzQrFkzfPjhh9rukAULFmDChAm4du0agoOD8fPPP8PS0hIA0KxZM2zbtg2zZs3C3Llz4e3tjU8++QQjRozQnmPNmjX48MMP8c477yApKQm1atXChx9+aIrqElExSEIIYepCEFH1Fh4ejhdffBHJyclwcnIydXGIqJxwTAgRERGZBIMQIiIiMgl2xxAREZFJsCWEiIiITIJBCBEREZkEgxAiIiIyCQYhREREZBIMQoiIiMgkGIQQERGRSTAIISIiIpNgEEJEREQm8f+LMSEATg25jwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 600x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* background: */\n",
       "    progress::-webkit-progress-bar {background-color: #CDCDCD; width: 100%;}\n",
       "    progress {background-color: #CDCDCD;}\n",
       "\n",
       "    /* value: */\n",
       "    progress::-webkit-progress-value {background-color: #00BFFF  !important;}\n",
       "    progress::-moz-progress-bar {background-color: #00BFFF  !important;}\n",
       "    progress {color: #00BFFF ;}\n",
       "\n",
       "    /* optional */\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #000000;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      <progress value='18' class='progress-bar-interrupted' max='100' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      18.00% [18/100 47:15<3:35:16][earlystopping]\n",
       "      <br>\n",
       "      \n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< val_loss without improvement in 5 epoch,early stopping >>>>>>\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "\n",
       "  <div id=\"df-9e002e9c-e7cd-428f-ac88-fca15eba0611\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>lr</th>\n",
       "      <th>val_loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.307877</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.162481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.154576</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.116936</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.120958</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.094219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.082410</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.084321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.079885</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.076271</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>0.069205</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.075862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>0.075754</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.071373</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>0.064483</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.071968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>0.068597</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.069226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>0.064511</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.068553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>0.072328</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.066686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>0.064761</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.067293</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>0.064022</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.065337</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>0.064710</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.065970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>0.067636</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.066208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>16</td>\n",
       "      <td>0.059463</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.066210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>17</td>\n",
       "      <td>0.065419</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.066222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>18</td>\n",
       "      <td>0.065364</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.066342</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9e002e9c-e7cd-428f-ac88-fca15eba0611')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "\n",
       "\n",
       "\n",
       "    <div id=\"df-567a89af-6d49-4108-947b-fbe066ab89ab\">\n",
       "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-567a89af-6d49-4108-947b-fbe066ab89ab')\"\n",
       "              title=\"Suggest charts.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "      </button>\n",
       "    </div>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "    background-color: #E8F0FE;\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: #1967D2;\n",
       "    height: 32px;\n",
       "    padding: 0 0 0 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: #E2EBFA;\n",
       "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: #174EA6;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "    background-color: #3B4455;\n",
       "    fill: #D2E3FC;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart:hover {\n",
       "    background-color: #434B5C;\n",
       "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "    fill: #FFFFFF;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "    <script>\n",
       "      async function quickchart(key) {\n",
       "        const containerElement = document.querySelector('#' + key);\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      }\n",
       "    </script>\n",
       "\n",
       "      <script>\n",
       "\n",
       "function displayQuickchartButton(domScope) {\n",
       "  let quickchartButtonEl =\n",
       "    domScope.querySelector('#df-567a89af-6d49-4108-947b-fbe066ab89ab button.colab-df-quickchart');\n",
       "  quickchartButtonEl.style.display =\n",
       "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "}\n",
       "\n",
       "        displayQuickchartButton(document);\n",
       "      </script>\n",
       "      <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-9e002e9c-e7cd-428f-ac88-fca15eba0611 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-9e002e9c-e7cd-428f-ac88-fca15eba0611');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "    epoch  train_loss        lr  val_loss\n",
       "0       1    0.307877  0.000002  0.162481\n",
       "1       2    0.154576  0.000002  0.116936\n",
       "2       3    0.120958  0.000002  0.094219\n",
       "3       4    0.082410  0.000002  0.084321\n",
       "4       5    0.079885  0.000002  0.076271\n",
       "5       6    0.069205  0.000002  0.075862\n",
       "6       7    0.075754  0.000002  0.071373\n",
       "7       8    0.064483  0.000002  0.071968\n",
       "8       9    0.068597  0.000002  0.069226\n",
       "9      10    0.064511  0.000002  0.068553\n",
       "10     11    0.072328  0.000002  0.066686\n",
       "11     12    0.064761  0.000002  0.067293\n",
       "12     13    0.064022  0.000002  0.065337\n",
       "13     14    0.064710  0.000002  0.065970\n",
       "14     15    0.067636  0.000002  0.066208\n",
       "15     16    0.059463  0.000002  0.066210\n",
       "16     17    0.065419  0.000002  0.066222\n",
       "17     18    0.065364  0.000002  0.066342"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 训练一百轮\n",
    "# patience=5：早停策略的参数，用于防止模型在训练过程中过拟合。如果在连续的5轮训练中，验证损失（或者其他的指定指标）都没有改善，那么训练将会提前停止\n",
    "# monitor='val_loss'：需要监控的指标：验证损失\n",
    "# mode='min'：早停策略的参数，指定了改进的方向。'min'意味着指标（这里是验证损失）的最小值被视为改进\n",
    "# mixed_precision='fp16'：这是一种混合精度训练的策略。在混合精度训练中，一部分张量的数据类型会被设为低精度（如半精度浮点数fp16），这样可以减少计算资源的需求，从而提高训练速度和效率\n",
    "keras_model.fit(train_data = dl_train,\n",
    "                val_data = dl_val,\n",
    "                epochs=100,patience=5,\n",
    "                monitor='val_loss',mode='min',\n",
    "                ckpt_path = ckpt_path,\n",
    "                mixed_precision='fp16'\n",
    "               )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "78zJ3Fs1Qwhy"
   },
   "source": [
    "## 验证模型训练后效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67,
     "referenced_widgets": [
      "1c6b2a8c8c8e48079676404272a85570",
      "3adf9df88e084929b37833fb8b59c540",
      "6f157a87750f4372b7b32623cacaf432",
      "1f3c2465ce1d4b908fe4686d49d4bd80",
      "0070d4654e5841d6a31967904a6a3276",
      "4d69c8d9ed794ffd8858709d94387c91",
      "026f56c85f67435b9d957ae2c5398aa5",
      "66173ad1451d4bdd9c1249497d703163",
      "998a73c402234335905ee58a33acf295",
      "75fb10c0f2a64ca4abb92eb8666f1d62",
      "3fccc3b552be4df8811bbd823ac31765"
     ]
    },
    "id": "uMbVvwO_LrU_",
    "outputId": "51ffe950-b6f9-46cd-d322-15b3a88edc48"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:accelerate.utils.modeling:The model weights are not tied. Please use the `tie_weights` method before using the `infer_auto_device` function.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1c6b2a8c8c8e48079676404272a85570",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/7 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from peft import PeftModel\n",
    "model = AutoModel.from_pretrained(\"THUDM/chatglm2-6b\",\n",
    "                                  load_in_8bit=False,\n",
    "                                  trust_remote_code=True,\n",
    "                                  device_map='auto')\n",
    "model = PeftModel.from_pretrained(model,ckpt_path)\n",
    "model = model.merge_and_unload() #合并lora权重"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "CgatzwphLyRP",
    "outputId": "b83a13b7-ba1a-404e-c415-d1e9df729712"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'差评'"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# def predict(text):\n",
    "#     response, history = model.chat(tokenizer, f\"{text} -> \", history=his,\n",
    "#     temperature=0.01)\n",
    "#     return response\n",
    "\n",
    "predict('死鬼，咋弄得这么有滋味呢')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UkbinpPRL4vT",
    "outputId": "584193f4-f148-4b82-ef27-21fbbe816725"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2000/2000 [04:58<00:00,  6.69it/s]\n"
     ]
    }
   ],
   "source": [
    "dftest = pd.read_parquet('dftest.parquet')\n",
    "preds = ['' for x in dftest['text']]\n",
    "# 重新走一遍对测试数据的验证流程\n",
    "from tqdm import tqdm\n",
    "for i in tqdm(range(len(dftest))):\n",
    "    text = dftest['text'].loc[i]\n",
    "    preds[i] = predict(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 143
    },
    "id": "HlosXnTeL78Z",
    "outputId": "3ebba7fc-aaa0-4789-81ee-d416340e1316"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\n",
       "  <div id=\"df-f9ae116f-3231-4630-9848-e8cc25a1aab2\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>pred</th>\n",
       "      <th>好评</th>\n",
       "      <th>好评，好评</th>\n",
       "      <th>差评</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tag</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>好评</th>\n",
       "      <td>869.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>124.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>差评</th>\n",
       "      <td>82.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>924.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f9ae116f-3231-4630-9848-e8cc25a1aab2')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "\n",
       "\n",
       "\n",
       "    <div id=\"df-d32a322f-86bc-4682-8e1b-b389f118f468\">\n",
       "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-d32a322f-86bc-4682-8e1b-b389f118f468')\"\n",
       "              title=\"Suggest charts.\"\n",
       "              style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "      </button>\n",
       "    </div>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "    background-color: #E8F0FE;\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: #1967D2;\n",
       "    height: 32px;\n",
       "    padding: 0 0 0 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: #E2EBFA;\n",
       "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: #174EA6;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "    background-color: #3B4455;\n",
       "    fill: #D2E3FC;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart:hover {\n",
       "    background-color: #434B5C;\n",
       "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "    fill: #FFFFFF;\n",
       "  }\n",
       "</style>\n",
       "\n",
       "    <script>\n",
       "      async function quickchart(key) {\n",
       "        const containerElement = document.querySelector('#' + key);\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      }\n",
       "    </script>\n",
       "\n",
       "      <script>\n",
       "\n",
       "function displayQuickchartButton(domScope) {\n",
       "  let quickchartButtonEl =\n",
       "    domScope.querySelector('#df-d32a322f-86bc-4682-8e1b-b389f118f468 button.colab-df-quickchart');\n",
       "  quickchartButtonEl.style.display =\n",
       "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "}\n",
       "\n",
       "        displayQuickchartButton(document);\n",
       "      </script>\n",
       "      <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-f9ae116f-3231-4630-9848-e8cc25a1aab2 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-f9ae116f-3231-4630-9848-e8cc25a1aab2');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "pred     好评  好评，好评     差评\n",
       "tag                      \n",
       "好评    869.0    1.0  124.0\n",
       "差评     82.0    NaN  924.0"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dftest['pred'] = preds\n",
    "dftest.pivot_table(index='tag',columns = 'pred',values='text',aggfunc='count')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "h_qU8ZolMUtY",
    "outputId": "5a39c73c-4e42-45a7-cc00-e7ab084ce288"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc= 0.8965\n"
     ]
    }
   ],
   "source": [
    "acc = len(dftest.query('tag==pred'))/len(dftest)\n",
    "print('acc=',acc)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3UDt0jdCWTgf"
   },
   "source": [
    "### Finetune 效果\n",
    "\n",
    "- 相比于没有Finetune只使用prompt进行任务的模型，准确率提升了2%左右，不算高"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UisngUY-MfJp"
   },
   "source": [
    "## 保存模型\n",
    "\n",
    "- 注意需要先装载你的Google Drive\n",
    "- 复制你的 Drive path 并基于 path 修改成下述的代码\n",
    "- run the code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NV4qIKBTMdhN",
    "outputId": "032401bf-95c2-488c-e136-eaeac29a45e6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('/content/drive/MyDrive/chatglm2-6b-waimai/tokenizer_config.json',\n",
       " '/content/drive/MyDrive/chatglm2-6b-waimai/special_tokens_map.json',\n",
       " '/content/drive/MyDrive/chatglm2-6b-waimai/tokenizer.model',\n",
       " '/content/drive/MyDrive/chatglm2-6b-waimai/added_tokens.json')"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.save_pretrained(\"/content/drive/MyDrive/chatglm2-6b-waimai\", max_shard_size='1GB')\n",
    "tokenizer.save_pretrained(\"/content/drive/MyDrive/chatglm2-6b-waimai\")"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "A100",
   "machine_shape": "hm",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0070d4654e5841d6a31967904a6a3276": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "022ee630affe4d909ffd5d8c839f6252": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "026f56c85f67435b9d957ae2c5398aa5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "07d7d5c813544d6c99694f3d903b78a4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "08107c31b06742d2abb80bf3e0626a61": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0823b03adb944aaf867be87e1b32fc95": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f719fc7d187248aaab5616d12048ccc9",
      "placeholder": "​",
      "style": "IPY_MODEL_40115d27540748a29589f8d5cd903bbc",
      "value": " 5970/6000 [00:07&lt;00:00, 826.91 examples/s]"
     }
    },
    "0ac5f7dab5324200be16659b1e950ca4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c6de6497352f4e6c99e2c24a9a2ffeab",
      "placeholder": "​",
      "style": "IPY_MODEL_07d7d5c813544d6c99694f3d903b78a4",
      "value": " 2000/2000 [00:00&lt;00:00, 6818.08 examples/s]"
     }
    },
    "1992b8e9af604d4684423094d2b625b0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_36202730c3b940bc9b741fa2c98cd6d8",
      "placeholder": "​",
      "style": "IPY_MODEL_e478d3c597d7447a9bc144f781919124",
      "value": "Map: 100%"
     }
    },
    "1c6b2a8c8c8e48079676404272a85570": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_3adf9df88e084929b37833fb8b59c540",
       "IPY_MODEL_6f157a87750f4372b7b32623cacaf432",
       "IPY_MODEL_1f3c2465ce1d4b908fe4686d49d4bd80"
      ],
      "layout": "IPY_MODEL_0070d4654e5841d6a31967904a6a3276"
     }
    },
    "1f3c2465ce1d4b908fe4686d49d4bd80": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_75fb10c0f2a64ca4abb92eb8666f1d62",
      "placeholder": "​",
      "style": "IPY_MODEL_3fccc3b552be4df8811bbd823ac31765",
      "value": " 7/7 [00:13&lt;00:00,  1.81s/it]"
     }
    },
    "321002d3fcc849579fa3693f41d809a0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "33207fc2cc5f4b6da3d9ac8f916a4a7d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_59acbc5323cd439b9b16a8df96b42fff",
      "placeholder": "​",
      "style": "IPY_MODEL_8f076715836f4d0dbb6e082a84178d2b",
      "value": "Map:  99%"
     }
    },
    "35da59a976ab401381793a70b087f849": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "36202730c3b940bc9b741fa2c98cd6d8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3adf9df88e084929b37833fb8b59c540": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4d69c8d9ed794ffd8858709d94387c91",
      "placeholder": "​",
      "style": "IPY_MODEL_026f56c85f67435b9d957ae2c5398aa5",
      "value": "Loading checkpoint shards: 100%"
     }
    },
    "3fccc3b552be4df8811bbd823ac31765": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "40115d27540748a29589f8d5cd903bbc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "420800e9429b45ee9b48f2d81388025a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_a0eb595e5fb1445a86a4fc4724196c60",
       "IPY_MODEL_687484ad16da42a986ca785667ace01c",
       "IPY_MODEL_b9d272b52bb149d8b07d32bcc7ee6b1a"
      ],
      "layout": "IPY_MODEL_7cd70c7b987147a8a65f42cf25e4578b"
     }
    },
    "451df9ca78c7409ab6bfd8891008d94a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4b58fd1b50804d16a9c723228e6bf9f5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4d56934947ec47b7a7ca161339ccd8a7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "4d69c8d9ed794ffd8858709d94387c91": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "59acbc5323cd439b9b16a8df96b42fff": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "61f8325c92ff4b9e9646d4593ff9aeeb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_1992b8e9af604d4684423094d2b625b0",
       "IPY_MODEL_fc08cc8ada404964afc78d14d362a307",
       "IPY_MODEL_0823b03adb944aaf867be87e1b32fc95"
      ],
      "layout": "IPY_MODEL_4d56934947ec47b7a7ca161339ccd8a7"
     }
    },
    "66173ad1451d4bdd9c1249497d703163": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "687484ad16da42a986ca785667ace01c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a70f769b20a6426dbf54f18d3df01089",
      "max": 7,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_8acd915a527b47188d1158f4bbb010e4",
      "value": 7
     }
    },
    "6f157a87750f4372b7b32623cacaf432": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_66173ad1451d4bdd9c1249497d703163",
      "max": 7,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_998a73c402234335905ee58a33acf295",
      "value": 7
     }
    },
    "75fb10c0f2a64ca4abb92eb8666f1d62": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7cd70c7b987147a8a65f42cf25e4578b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8036903c231f4da594e4735ef12e4e9b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_451df9ca78c7409ab6bfd8891008d94a",
      "placeholder": "​",
      "style": "IPY_MODEL_81afe9b91f5a4cbea3e1fd5563bb509e",
      "value": " 6000/6000 [00:00&lt;00:00, 7264.14 examples/s]"
     }
    },
    "81afe9b91f5a4cbea3e1fd5563bb509e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "85cd0c14acf440b2b426fec7ec8bf6a7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "85fec979409f4e1c948ef76d1715cd85": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d1feb1ad37024a33a209fad97394a4ba",
      "placeholder": "​",
      "style": "IPY_MODEL_f4ae29eb498a4db1bc8c529ade45929e",
      "value": "Filter: 100%"
     }
    },
    "89f667e3508c4691a770c1325e80b9aa": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c9065d85291a46148eadb8faf3b8d088",
      "max": 2000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_e638631281654e50b855a5daa7ee7c99",
      "value": 2000
     }
    },
    "8acd915a527b47188d1158f4bbb010e4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "8f076715836f4d0dbb6e082a84178d2b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8fb6b350d38941038fe7a10673c42f49": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "998a73c402234335905ee58a33acf295": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "99e22f30c52847aa97bf1d31c46a29eb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_321002d3fcc849579fa3693f41d809a0",
      "max": 2000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_85cd0c14acf440b2b426fec7ec8bf6a7",
      "value": 2000
     }
    },
    "9b5410c75f4e49ec926f4d37528321b2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "9bc3d35d44864ae4b32fb4387492af30": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9d1e00e2347e42d0b8704e562cd51da6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_08107c31b06742d2abb80bf3e0626a61",
      "placeholder": "​",
      "style": "IPY_MODEL_be14f527c0374ca89def84c3cb778c4d",
      "value": " 1979/2000 [00:02&lt;00:00, 813.90 examples/s]"
     }
    },
    "a0eb595e5fb1445a86a4fc4724196c60": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_35da59a976ab401381793a70b087f849",
      "placeholder": "​",
      "style": "IPY_MODEL_cba1a27cd05340f191e9093a0098c6f7",
      "value": "Loading checkpoint shards: 100%"
     }
    },
    "a6b108bd8a734774992f7cb89e02a9c6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "a70f769b20a6426dbf54f18d3df01089": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a874214dde164270a43c60f0bfee959e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_022ee630affe4d909ffd5d8c839f6252",
      "max": 6000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_9b5410c75f4e49ec926f4d37528321b2",
      "value": 6000
     }
    },
    "aa12dfc3bdbe4c79b2164d1b1f8c65c9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "b00a3042be6e4462a982072b10f33d06": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "b9d272b52bb149d8b07d32bcc7ee6b1a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_cf48a73ae8034620a8c946e92724081f",
      "placeholder": "​",
      "style": "IPY_MODEL_e0da03b6f9b943e69095c5fba3a582f3",
      "value": " 7/7 [00:10&lt;00:00,  1.40s/it]"
     }
    },
    "be14f527c0374ca89def84c3cb778c4d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c38dba11819c4531ad37f023c427620e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_33207fc2cc5f4b6da3d9ac8f916a4a7d",
       "IPY_MODEL_99e22f30c52847aa97bf1d31c46a29eb",
       "IPY_MODEL_9d1e00e2347e42d0b8704e562cd51da6"
      ],
      "layout": "IPY_MODEL_a6b108bd8a734774992f7cb89e02a9c6"
     }
    },
    "c603c024236f4f98a2e16402e08b61e0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9bc3d35d44864ae4b32fb4387492af30",
      "placeholder": "​",
      "style": "IPY_MODEL_8fb6b350d38941038fe7a10673c42f49",
      "value": "Filter: 100%"
     }
    },
    "c6de6497352f4e6c99e2c24a9a2ffeab": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c9065d85291a46148eadb8faf3b8d088": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "cba1a27cd05340f191e9093a0098c6f7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "cf48a73ae8034620a8c946e92724081f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d1feb1ad37024a33a209fad97394a4ba": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d2fdaf73aab94e0b9d61a24310a9eccc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_c603c024236f4f98a2e16402e08b61e0",
       "IPY_MODEL_a874214dde164270a43c60f0bfee959e",
       "IPY_MODEL_8036903c231f4da594e4735ef12e4e9b"
      ],
      "layout": "IPY_MODEL_aa12dfc3bdbe4c79b2164d1b1f8c65c9"
     }
    },
    "e0da03b6f9b943e69095c5fba3a582f3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e478d3c597d7447a9bc144f781919124": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e638631281654e50b855a5daa7ee7c99": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "f05cfbd11828456096260430e33e1df0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "f4ae29eb498a4db1bc8c529ade45929e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f4d3233d514841739d1b5cad0371b0cc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_85fec979409f4e1c948ef76d1715cd85",
       "IPY_MODEL_89f667e3508c4691a770c1325e80b9aa",
       "IPY_MODEL_0ac5f7dab5324200be16659b1e950ca4"
      ],
      "layout": "IPY_MODEL_f05cfbd11828456096260430e33e1df0"
     }
    },
    "f719fc7d187248aaab5616d12048ccc9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fc08cc8ada404964afc78d14d362a307": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4b58fd1b50804d16a9c723228e6bf9f5",
      "max": 6000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_b00a3042be6e4462a982072b10f33d06",
      "value": 6000
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}