examples/BaiChuan13B_QLoRA——transformers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4e7bef94-e459-4140-b533-cd6c188722b0",
   "metadata": {},
   "source": [
    "# Baichuan-13B 保姆级微调范例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdbf35ed-b6c1-477f-89bb-c059573bffcf",
   "metadata": {},
   "source": [
    "😋😋公众号算法美食屋后台回复关键词：**torchkeras**，获取本文notebook源代码和数据集下载链接。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccbccebf-138c-445d-b386-9e4add0ad3ed",
   "metadata": {},
   "source": [
    "干货预警：这可能是你能够找到的最容易懂的，最完整的，适用于各种NLP任务的Baichuan-13B-Chat的finetune教程~\n",
    "\n",
    "Baichuan-13B是百川智能于2023年7月11日发布的开源中英双语LLM，各项指标经评测在开源LLM中同尺寸模型中位居前列。\n",
    "\n",
    "Baichuan-13B包括Baichuan-13B-Base和Baichuan-13B-chat两个不同模型。前者仅仅是预训练模型，后者在前者基础上增加了SFT,RLHF等偏好对齐过程。\n",
    "\n",
    "本范例微调的模型是Baichuan-13B-Chat，我们使用非常简单的，外卖评论数据集来实施微调，对一段外卖评论区分是好评还是差评。\n",
    "\n",
    "可以发现，经过微调后的模型，相比直接 3-shot-prompt 可以取得明显更好的效果(0.89->0.90)。\n",
    "\n",
    "虽然Baichuan-13B-Chat是一个百亿级的LLM，但由于我们使用非常节约显存的QLoRA微调算法，具备32G左右显存的GPU即可实施本过程。\n",
    "\n",
    "值得注意的是，尽管我们以文本分类任务为例，实际上，任何NLP任务，例如，命名实体识别，翻译，聊天对话等等，都可以通过加上合适的上下文，转换成一个对话问题，并针对我们的使用场景，设计出合适的数据集来微调Baichuan-13B-Chat.\n",
    "\n",
    "\n",
    "注，本教程是 ChatGLM2-6b保姆级微调范例 的兄弟版本~\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ee10aa6-bd22-4014-8fa4-7f5a49c884b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "#安装环境\n",
    "\n",
    "#baichuan-13b-chat\n",
    "#!pip install 'transformers==4.30.2'\n",
    "#!pip install  -U transformers_stream_generator\n",
    "\n",
    "\n",
    "#finetune\n",
    "#!pip install datasets\n",
    "#!pip install git+https://github.com/huggingface/accelerate\n",
    "#!pip install  git+https://github.com/huggingface/peft\n",
    "#!pip install  git+https://github.com/lyhue1991/torchkeras \n",
    "#!pip install 'bitsandbytes==0.39.1' #4bit量化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "237a11c4-0f74-4f67-9971-684cd0903342",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25G\t../baichuan-13b\n"
     ]
    }
   ],
   "source": [
    "!du -s -h ../baichuan-13b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eae3a0b3-bf03-46bd-bace-330665645f9a",
   "metadata": {},
   "source": [
    "## 〇，预训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e633e3b-e4cc-48d4-8792-ea269c7396f6",
   "metadata": {},
   "source": [
    "我们需要从 https://huggingface.co/baichuan-inc/Baichuan-13B-Chat 下载baichuan-13b-chat的模型。\n",
    "\n",
    "国内可能速度会比较慢，总共有25个G左右，网速不太好的话，大概可能需要两到三个小时。\n",
    "\n",
    "如果网络不稳定，也可以手动从这个页面一个一个下载全部文件然后放置到 一个文件夹中例如 'baichuan-13b' 以便读取。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9c0f2e17-7ed1-42e8-81f8-63a7bbeab31e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e200470d-5cec-44fd-a733-b26b7ff43a63",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-07-19 20:58:18,589] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)\n",
      "\n",
      "===================================BUG REPORT===================================\n",
      "Welcome to bitsandbytes. For bug reports, please run\n",
      "\n",
      "python -m bitsandbytes\n",
      "\n",
      " and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
      "================================================================================\n",
      "bin /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so\n",
      "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...\n",
      "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0\n",
      "CUDA SETUP: Highest compute capability among GPUs detected: 8.0\n",
      "CUDA SETUP: Detected CUDA version 116\n",
      "CUDA SETUP: Loading binary /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d3facd0d972f4d32aeea744806a96fb6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, AutoModel, BitsAndBytesConfig\n",
    "from transformers.generation.utils import GenerationConfig\n",
    "import torch.nn as nn\n",
    "\n",
    "\n",
    "#使用QLoRA引入的 NF4量化数据类型以节约显存\n",
    "model_name_or_path ='../baichuan-13b' #远程 'baichuan-inc/Baichuan-13B-Chat'\n",
    "\n",
    "bnb_config=BitsAndBytesConfig(\n",
    "            load_in_4bit=True,\n",
    "            bnb_4bit_compute_dtype=torch.float16,\n",
    "            bnb_4bit_use_double_quant=True,\n",
    "            bnb_4bit_quant_type=\"nf4\",\n",
    "            llm_int8_threshold=6.0,\n",
    "            llm_int8_has_fp16_weight=False,\n",
    "        )\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "   model_name_or_path, trust_remote_code=True)\n",
    "\n",
    "model = AutoModelForCausalLM.from_pretrained(model_name_or_path,\n",
    "                quantization_config=bnb_config,\n",
    "                trust_remote_code=True) \n",
    "\n",
    "model.generation_config = GenerationConfig.from_pretrained(model_name_or_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "76722209-62a4-4e17-a672-9383791cd014",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "乔戈里峰。世界第二高峰———乔戈里峰海拔高度：8610米坐标：35.5°n,76.5°e乔戈里，蒙古语意为“高大雄伟”，它位于喀喇昆仑山脉的中部。\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import clear_output \n",
    "messages = []\n",
    "messages.append({\"role\": \"user\",\n",
    "                 \"content\": \"世界上第二高的山峰是哪座?\"})\n",
    "response = model.chat(tokenizer,messages=messages,stream=True)\n",
    "for res in response:\n",
    "    print(res)\n",
    "    clear_output(wait=True)\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b93cd14e-35c2-4e15-bf5f-1b757aefc1a6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a4045ef-14c7-4eb3-b577-a70e81d23ca7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c754a8e9-c9d6-49c4-8a43-daafc8374129",
   "metadata": {},
   "source": [
    "下面我们设计一个3-shot-prompt方法，使用外卖数据集测试一下BaiChuan13b的文本分类能力。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81c1e15f-2217-4a76-8caa-945362f98317",
   "metadata": {},
   "outputs": [],
   "source": [
    "prefix = \"\"\"外卖评论文本分类任务:\n",
    "下面是一些范例:\n",
    "\n",
    "味道真不错 -> 好评\n",
    "太辣了，吃不下都  -> 差评\n",
    "\n",
    "请对下述评论进行分类。返回'好评'或者'差评'。\n",
    "\"\"\"\n",
    "\n",
    "def get_prompt(text):\n",
    "    return prefix+text+' -> '\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9974c379-3bf3-4089-b656-3d0861b1e76a",
   "metadata": {},
   "outputs": [],
   "source": [
    "messages  = [{\"role\": \"user\", \"content\": get_prompt('味道不错，下次再来')}]\n",
    "response = model.chat(tokenizer, messages)\n",
    "print(response)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0b5ea5e0-70a0-4e72-8452-485819157bf2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "好评\n"
     ]
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "96c7d5d3-4ec0-47f4-8592-6cc77e7d54a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'role': 'user', 'content': \"外卖评论文本分类任务:\\n下面是一些范例:\\n\\n味道真不错 -> 好评\\n太辣了，吃不下都  -> 差评\\n\\n请对下述评论进行分类。返回'好评'或者'差评'。\\n味道不错，下次再来 -> \"}, {'role': 'assistant', 'content': '好评'}]\n"
     ]
    }
   ],
   "source": [
    "messages = messages+[{\"role\": \"assistant\", \"content\": response}]\n",
    "print(messages)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a078a521-b3c3-40c8-b7a5-9505a9cbc80e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_message(prompt,response):\n",
    "    return [{\"role\": \"user\", \"content\": f'{prompt} -> '},\n",
    "            {\"role\": \"assistant\", \"content\": response}]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "30247e96-0f93-4c05-bc2d-8c836db05dd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "messages.extend(get_message('太贵了','差评'))\n",
    "messages.extend(get_message('非常快，味道好','好评'))\n",
    "messages.extend(get_message('这么咸，真的是醉了','差评'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9e1c5fdc-afd9-44d9-a96b-8591af55801c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'role': 'user',\n",
       "  'content': \"外卖评论文本分类任务:\\n下面是一些范例:\\n\\n味道真不错 -> 好评\\n太辣了，吃不下都  -> 差评\\n\\n请对下述评论进行分类。返回'好评'或者'差评'。\\n味道不错，下次再来 -> \"},\n",
       " {'role': 'assistant', 'content': '好评'},\n",
       " {'role': 'user', 'content': '太贵了 -> '},\n",
       " {'role': 'assistant', 'content': '差评'},\n",
       " {'role': 'user', 'content': '非常快，味道好 -> '},\n",
       " {'role': 'assistant', 'content': '好评'},\n",
       " {'role': 'user', 'content': '这么咸，真的是醉了 -> '},\n",
       " {'role': 'assistant', 'content': '差评'}]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "messages "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "bdaf6e0f-fd17-40e1-b790-78593e382d93",
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(text,temperature=0.01):\n",
    "    model.generation_config.temperature=temperature\n",
    "    response = model.chat(tokenizer, \n",
    "                          messages = messages+[{'role':'user','content':f'{text} -> '}])\n",
    "    return response\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "d2dffa85-2932-4bee-b60d-965eedd88604",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'好评'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict('死鬼，咋弄得这么有滋味呢！') #惊喜，居然预测对了"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "659f9d62-34bb-4589-9bb5-f25090199775",
   "metadata": {},
   "source": [
    "我们拿外卖数据集来测试一下未经微调，预训练模型的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "cadf4887-a715-4e36-9e80-419619f4b983",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np \n",
    "import datasets \n",
    "from tqdm import tqdm \n",
    "\n",
    "#数据集分割\n",
    "# df = pd.read_csv(\"data/waimai_10k.csv\")\n",
    "\n",
    "# df['tag'] = df['label'].map({0:'差评',1:'好评'})\n",
    "# df = df.rename({'review':'text'},axis = 1)\n",
    "\n",
    "# dfgood = df.query('tag==\"好评\"')\n",
    "# dfbad = df.query('tag==\"差评\"').head(len(dfgood)) #采样部分差评，让好评差评平衡\n",
    "# df = pd.concat([dfgood,dfbad])\n",
    "\n",
    "# ds_dic = datasets.Dataset.from_pandas(df).train_test_split(\n",
    "#     test_size = 2000,shuffle=True, seed = 43)\n",
    "# dftrain = ds_dic['train'].to_pandas()\n",
    "# dftest = ds_dic['test'].to_pandas()\n",
    "# dftrain.to_parquet('../data/dftrain.parquet')\n",
    "# dftest.to_parquet('../data/dftest.parquet')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f41ca74-cb57-4f4d-a74a-66ad39262224",
   "metadata": {},
   "outputs": [],
   "source": [
    "#数据集加载\n",
    "dftrain = pd.read_parquet('../data/dftrain.parquet')[['text','label','tag']]\n",
    "dftest = pd.read_parquet('../data/dftest.parquet')[['text','label','tag']]\n",
    "ds_train,ds_val = datasets.Dataset.from_pandas(dftrain).train_test_split(\n",
    "    test_size=1000,seed=42).values()\\\n",
    "\n",
    "dftrain,dfval = ds_train.to_pandas(), ds_val.to_pandas() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "44fe3c7a-2857-478c-9041-42713e4b5b9b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2000/2000 [10:28<00:00,  3.18it/s]\n"
     ]
    }
   ],
   "source": [
    "dftest['pred'] = [predict(text) for text in tqdm(dftest['text'])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "82b2d2ab-ef2e-47c3-b7de-5b1a1c32877a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1785, 4)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dftest.query('tag==pred').shape  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "3dfb5ddd-8573-4c38-b746-68adea5aeea6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc =  0.8925\n"
     ]
    }
   ],
   "source": [
    "print('acc = ',len(dftest.query('tag==pred'))/len(dftest))   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "7015a6a2-451b-4db8-9c22-31f24bb68ae4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>tag</th>\n",
       "      <th>好评</th>\n",
       "      <th>差评</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pred</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>好评</th>\n",
       "      <td>853.0</td>\n",
       "      <td>74.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>差评</th>\n",
       "      <td>140.0</td>\n",
       "      <td>932.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>无法判断</th>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "tag      好评     差评\n",
       "pred              \n",
       "好评    853.0   74.0\n",
       "差评    140.0  932.0\n",
       "无法判断    1.0    NaN"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dftest.pivot_table(index='pred',columns = 'tag',\n",
    "                   values='text',aggfunc='count')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c083eecf-6bd8-4450-9413-d10742eaffc3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "09be19b6-615f-4488-bd27-6dc08a015105",
   "metadata": {},
   "source": [
    "## 一，准备数据"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23d48441-441e-41c0-9442-7aeb8da003c3",
   "metadata": {},
   "source": [
    "我们仿照百川模型的 `model._build_chat_input` 方法来进行token编码，同时把需要学习的内容添加label."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d442edb-dcd5-46cf-91a9-3203bccbe9a4",
   "metadata": {},
   "source": [
    "### 1，token编码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "9fcd2921-ca7d-43c6-b49b-e4e6a65ed37d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "\n",
    "#将messages编码成 token, 同时返回labels, 该函数适用于多轮对话数据\n",
    "#注意baichuan-13b通过插入tokenizer.user_token_id和tokenizer.assistant_token_id 来区分用户和机器人会话内容\n",
    "\n",
    "# reference@ model._build_chat_input?\n",
    "def build_chat_input(messages, model=model,\n",
    "                     tokenizer=tokenizer, \n",
    "                     max_new_tokens: int=0):\n",
    "    max_new_tokens = max_new_tokens or model.generation_config.max_new_tokens\n",
    "    max_input_tokens = model.config.model_max_length - max_new_tokens\n",
    "    max_input_tokens = max(model.config.model_max_length // 2, max_input_tokens)\n",
    "    \n",
    "    total_input, round_input, total_label, round_label = [], [], [], []\n",
    "    \n",
    "    for i, message in enumerate(messages[::-1]):\n",
    "        content_tokens = tokenizer.encode(message['content'])\n",
    "        if message['role'] == 'user':\n",
    "            round_input = [model.generation_config.user_token_id] + content_tokens + round_input\n",
    "            round_label = [-100]+[-100 for _ in content_tokens]+ round_label\n",
    "            \n",
    "            if total_input and len(total_input) + len(round_input) > max_input_tokens:\n",
    "                break\n",
    "            else:\n",
    "                total_input = round_input + total_input\n",
    "                total_label = round_label + total_label\n",
    "                if len(total_input) >= max_input_tokens:\n",
    "                    break\n",
    "                else:\n",
    "                    round_input = []\n",
    "                    round_label = []\n",
    "                    \n",
    "        elif message['role'] == 'assistant':\n",
    "            round_input = [\n",
    "                model.generation_config.assistant_token_id\n",
    "            ] + content_tokens + [\n",
    "                model.generation_config.eos_token_id\n",
    "            ] + round_input\n",
    "\n",
    "            round_label = [\n",
    "                -100\n",
    "            ] + content_tokens + [\n",
    "                model.generation_config.eos_token_id\n",
    "            ]+ round_label\n",
    "        else:\n",
    "            raise ValueError(f\"message role not supported yet: {message['role']}\")\n",
    "            \n",
    "    total_input = total_input[-max_input_tokens:]  # truncate left\n",
    "    total_label = total_label[-max_input_tokens:]\n",
    "    \n",
    "    total_input.append(model.generation_config.assistant_token_id)\n",
    "    total_label.append(-100)\n",
    "    \n",
    "    return total_input,total_label\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c2d9290-53a7-4ae3-a1ca-d972512c72cb",
   "metadata": {},
   "source": [
    "### 2，做数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "b9a76500-01db-4c34-b2ba-6bc1c56233d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset,DataLoader \n",
    "class MyDataset(Dataset):\n",
    "    def __init__(self,df,\n",
    "                 prefix=prefix\n",
    "                ):\n",
    "        self.df = df \n",
    "        self.prefix=prefix\n",
    "        \n",
    "    def __len__(self):\n",
    "        return len(self.df)\n",
    "        \n",
    "    def get_samples(self,index):\n",
    "        samples = []\n",
    "        d = dict(self.df.iloc[index])\n",
    "        samples.append(d)\n",
    "        return samples\n",
    "    \n",
    "    def get_messages(self,index):\n",
    "        samples = self.get_samples(index)\n",
    "        messages = []\n",
    "        for i,d in enumerate(samples):\n",
    "            if i==0:\n",
    "                messages.append({'role':'user','content':self.prefix+d['text']+' -> '})\n",
    "            else:\n",
    "                messages.append({'role':'user','content':d['text']+' -> '})\n",
    "            \n",
    "            messages.append({'role':'assistant','content':d['tag']})\n",
    "        return messages\n",
    "        \n",
    "    def __getitem__(self,index):\n",
    "        messages = self.get_messages(index)\n",
    "        input_ids, labels = build_chat_input(messages)\n",
    "        return {'input_ids':input_ids,'labels':labels}\n",
    "\n",
    "    def show_sample(self,index):\n",
    "        samples = self.get_samples(index)\n",
    "        print(samples)\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "6cc385bf-0103-4720-a4bd-42744aa76f9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_train = MyDataset(dftrain)\n",
    "ds_val = MyDataset(dfval)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "f19fe484-bc8b-44dd-8b26-d1a8714c0990",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'text': '送的慢1个多小时，卷饼特别干，没有老北京鸡肉卷好吃，因为里面没有酱，而且肉不太好，感觉不一定是肘子肉，不会再吃了', 'label': 0, 'tag': '差评'}]\n"
     ]
    }
   ],
   "source": [
    "ds_train.show_sample(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "0fcb4c4b-a4c7-4439-92e0-61f2148c13d7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input_ids': [195, 31106, 17924, 9929, 31271, 31267, 10732, 5934, 31143, 5, 7810, 31161, 2719, 31751, 31530, 31143, 5, 5, 10685, 31426, 6891, 11531, 31106, 24275, 5, 31605, 32747, 31172, 72, 31604, 18444, 31228, 99, 3817, 31106, 31965, 31784, 5, 5, 31488, 31209, 31214, 32193, 9929, 1696, 10732, 73, 17965, 31155, 24275, 31155, 2818, 31155, 31965, 31784, 31155, 73, 5, 31818, 31135, 32044, 53, 31179, 31206, 4237, 72, 32303, 33063, 3450, 31621, 72, 1449, 31367, 3659, 32180, 31915, 32303, 22823, 72, 1863, 7515, 1449, 33029, 72, 3421, 31915, 13204, 31213, 72, 3994, 31166, 20975, 34318, 31225, 31915, 72, 2753, 31513, 18392, 11531, 31106, 196, 31106, 31965, 31784, 2, 196], 'labels': [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 31106, 31965, 31784, 2, -100]}\n"
     ]
    }
   ],
   "source": [
    "print(ds_train[0]) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc80b2ee-7d70-41a7-8cf7-2bb5fdfff151",
   "metadata": {},
   "source": [
    "### 3，创建管道"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "e43aaccc-863e-4794-8cb5-683d5a727ada",
   "metadata": {},
   "outputs": [],
   "source": [
    "def data_collator(examples: list):\n",
    "    len_ids = [len(example[\"input_ids\"]) for example in examples]\n",
    "    longest = max(len_ids) #之后按照batch中最长的input_ids进行padding\n",
    "    \n",
    "    input_ids = []\n",
    "    labels_list = []\n",
    "    \n",
    "    for length, example in sorted(zip(len_ids, examples), key=lambda x: -x[0]):\n",
    "        ids = example[\"input_ids\"]\n",
    "        labs = example[\"labels\"]\n",
    "        \n",
    "        ids = ids + [tokenizer.pad_token_id] * (longest - length)\n",
    "        labs = labs + [-100] * (longest - length)\n",
    "        \n",
    "        input_ids.append(torch.LongTensor(ids))\n",
    "        labels_list.append(torch.LongTensor(labs))\n",
    "          \n",
    "    input_ids = torch.stack(input_ids)\n",
    "    labels = torch.stack(labels_list)\n",
    "    return {\n",
    "        \"input_ids\": input_ids,\n",
    "        \"labels\": labels,\n",
    "    }\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "698aa807-e05a-42a8-b392-747982363075",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "dl_train = torch.utils.data.DataLoader(ds_train,num_workers=2,batch_size=4,\n",
    "                                       pin_memory=True,shuffle=True,\n",
    "                                       collate_fn = data_collator)\n",
    "\n",
    "dl_val = torch.utils.data.DataLoader(ds_val,num_workers=2,batch_size=4,\n",
    "                                    pin_memory=True,shuffle=False,\n",
    "                                     collate_fn = data_collator)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "db52d785-10de-4465-8fc2-9eaccaf3387d",
   "metadata": {},
   "outputs": [],
   "source": [
    "for batch in dl_train:\n",
    "    break "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "35002105-b8e8-4d48-a8a2-cbd4ceaa53ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "#试跑一个batch\n",
    "out = model(**batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "c721189c-6443-4e10-b923-327a0beba2fd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.0352, dtype=torch.float16, grad_fn=<ToCopyBackward0>)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out.loss "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "7cac7d4a-a37b-4c30-8a7d-be552d8e67ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1250"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dl_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "2eb0f563-61b0-4576-a3b1-08aa4b62e52f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#采样300个batch作为一个epoch，便于较快验证\n",
    "dl_train.size = 300"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37fd7e26-985a-4392-ba0d-acb254064677",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5aa1deea-3fc4-4051-80e9-89de248d3107",
   "metadata": {},
   "source": [
    "## 二，定义模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "644837f5-bbc4-4f4f-a5f0-1f763e36f373",
   "metadata": {},
   "source": [
    "下面我们将使用QLoRA(实际上用的是量化的AdaLoRA）算法来微调Baichuan-13b模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "b7dbbece-2e50-4166-9e43-ea35d04af856",
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import get_peft_config, get_peft_model, TaskType\n",
    "model.supports_gradient_checkpointing = True  #\n",
    "model.gradient_checkpointing_enable()\n",
    "model.enable_input_require_grads()\n",
    "\n",
    "model.config.use_cache = False  # silence the warnings. Please re-enable for inference!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "fc66d6a4-77ff-490c-a3c0-913a9316eb13",
   "metadata": {},
   "outputs": [],
   "source": [
    "import bitsandbytes as bnb \n",
    "def find_all_linear_names(model):\n",
    "    \"\"\"\n",
    "    找出所有全连接层，为所有全连接添加adapter\n",
    "    \"\"\"\n",
    "    cls = bnb.nn.Linear4bit\n",
    "    lora_module_names = set()\n",
    "    for name, module in model.named_modules():\n",
    "        if isinstance(module, cls):\n",
    "            names = name.split('.')\n",
    "            lora_module_names.add(names[0] if len(names) == 1 else names[-1])\n",
    "\n",
    "    if 'lm_head' in lora_module_names:  # needed for 16-bit\n",
    "        lora_module_names.remove('lm_head')\n",
    "    return list(lora_module_names)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "587a53eb-ba75-40a6-83e6-1ad2684738f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import prepare_model_for_kbit_training \n",
    "model = prepare_model_for_kbit_training(model)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "ffbd01ad-2667-4c38-8ce7-62ce293e759e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['down_proj', 'o_proj', 'up_proj', 'W_pack', 'gate_proj']\n"
     ]
    }
   ],
   "source": [
    "lora_modules = find_all_linear_names(model)\n",
    "print(lora_modules) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "e5b68e33-3c18-4044-9554-c453b17d4fef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trainable params: 41,843,040 || all params: 7,002,181,160 || trainable%: 0.5975715144165165\n"
     ]
    }
   ],
   "source": [
    "from peft import AdaLoraConfig\n",
    "peft_config = AdaLoraConfig(\n",
    "    task_type=TaskType.CAUSAL_LM, inference_mode=False,\n",
    "    r=64,\n",
    "    lora_alpha=16, lora_dropout=0.05,\n",
    "    target_modules= lora_modules\n",
    ")\n",
    "\n",
    "peft_model = get_peft_model(model, peft_config)\n",
    "\n",
    "peft_model.is_parallelizable = True\n",
    "peft_model.model_parallel = True\n",
    "peft_model.print_trainable_parameters()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "372d96e5-f5c4-4a21-a09b-825df8563399",
   "metadata": {},
   "outputs": [],
   "source": [
    "out = peft_model.forward(**batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "7b1c036a-c466-45a8-816e-e93da6810c87",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.0347, grad_fn=<ToCopyBackward0>)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74ddcd08-851b-42e0-a1ac-db530e728aee",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2ad2444d-2c9c-4048-958c-ae48d9590dfd",
   "metadata": {},
   "source": [
    "## 三，训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "9317cd85-b886-4153-af53-42c83921525a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchkeras import KerasModel \n",
    "from accelerate import Accelerator \n",
    "\n",
    "class StepRunner:\n",
    "    def __init__(self, net, loss_fn, accelerator=None, stage = \"train\", metrics_dict = None, \n",
    "                 optimizer = None, lr_scheduler = None\n",
    "                 ):\n",
    "        self.net,self.loss_fn,self.metrics_dict,self.stage = net,loss_fn,metrics_dict,stage\n",
    "        self.optimizer,self.lr_scheduler = optimizer,lr_scheduler\n",
    "        self.accelerator = accelerator if accelerator is not None else Accelerator() \n",
    "        if self.stage=='train':\n",
    "            self.net.train() \n",
    "        else:\n",
    "            self.net.eval()\n",
    "    \n",
    "    def __call__(self, batch):\n",
    "        \n",
    "        #loss\n",
    "        with self.accelerator.autocast():\n",
    "            loss = self.net.forward(**batch)[0]\n",
    "\n",
    "        #backward()\n",
    "        if self.optimizer is not None and self.stage==\"train\":\n",
    "            self.accelerator.backward(loss)\n",
    "            if self.accelerator.sync_gradients:\n",
    "                self.accelerator.clip_grad_norm_(self.net.parameters(), 1.0)\n",
    "            self.optimizer.step()\n",
    "            if self.lr_scheduler is not None:\n",
    "                self.lr_scheduler.step()\n",
    "            self.optimizer.zero_grad()\n",
    "            \n",
    "        all_loss = self.accelerator.gather(loss).sum()\n",
    "        \n",
    "        #losses (or plain metrics that can be averaged)\n",
    "        step_losses = {self.stage+\"_loss\":all_loss.item()}\n",
    "        \n",
    "        #metrics (stateful metrics)\n",
    "        step_metrics = {}\n",
    "        \n",
    "        if self.stage==\"train\":\n",
    "            if self.optimizer is not None:\n",
    "                step_metrics['lr'] = self.optimizer.state_dict()['param_groups'][0]['lr']\n",
    "            else:\n",
    "                step_metrics['lr'] = 0.0\n",
    "        return step_losses,step_metrics\n",
    "    \n",
    "KerasModel.StepRunner = StepRunner \n",
    "\n",
    "#仅仅保存QLora可训练参数\n",
    "def save_ckpt(self, ckpt_path='checkpoint', accelerator = None):\n",
    "    unwrap_net = accelerator.unwrap_model(self.net)\n",
    "    unwrap_net.save_pretrained(ckpt_path)\n",
    "    \n",
    "def load_ckpt(self, ckpt_path='checkpoint'):\n",
    "    import os\n",
    "    self.net.load_state_dict(\n",
    "        torch.load(os.path.join(ckpt_path,'adapter_model.bin')),strict =False)\n",
    "    self.from_scratch = False\n",
    "    \n",
    "KerasModel.save_ckpt = save_ckpt \n",
    "KerasModel.load_ckpt = load_ckpt \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "353a96d4-0fef-419a-a345-f5a5342ab8cc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "cbed7bbe-10a7-4132-8d02-ead60b4d59ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = bnb.optim.adamw.AdamW(peft_model.parameters(),\n",
    "                                  lr=6e-05,is_paged=True)  #'paged_adamw'\n",
    "keras_model = KerasModel(peft_model,loss_fn =None,\n",
    "        optimizer=optimizer) \n",
    "ckpt_path = 'baichuan13b_waimai'\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "131ba617-d6e6-4bf7-9259-daaa113ada8f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< ⚡️ cuda is used >>>>>>\u001b[0m\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiEAAAGJCAYAAABcsOOZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACK0UlEQVR4nO3dd1hTZ/sH8O9JgLBB9hSciIhQF6KvWisVR1XEgavO1g61Wlpbsa1afSttHT9ntfatVVu1LrStWlu1UhfuiSIuFES2DAFZyfP745BIIIEAWcD9ua5ckJPnnPMckpA7z7gfjjHGQAghhBCiZQJdV4AQQgghTRMFIYQQQgjRCQpCCCGEEKITFIQQQgghRCcoCCGEEEKITlAQQgghhBCdoCCEEEIIITpBQQghhBBCdIKCEEIIIYToBAUhROMWLVoEjuOQmZmp66pozaNHj8BxHLZs2aLRfUjD8+2336Jdu3aQSCS6rorW6OP/gHnz5iEgIEDX1WjyKAghjdbSpUtx4MABXVeDVJKTk4Pp06fD3t4eZmZm6Nu3L65cuaLy/nFxcRgwYADMzc1hY2ODN998ExkZGXJlpB96ym5nzpxReOzS0lK0b98eHMdh+fLlco89ffoUEyZMgJeXFywsLGBtbY1u3bph69atUHX1i7y8PHzzzTf49NNPIRDU/O+3qKgIpaWlKh27MYuPj8eHH36IHj16wNjYGBzH4dGjR0rL//777+jUqROMjY3RvHlzLFy4EGVlZXJl5syZg+vXr+P333/XcO1JdQx0XQFCNGXp0qUYOXIkQkJCdF0VUk4ikWDw4MG4fv065s6dCzs7O3z33Xd49dVXcfnyZbRp06ba/Z88eYLevXvDysoKS5cuRX5+PpYvX46bN2/iwoULMDIyAgCEhoaidevWVfafP38+8vPz0bVrV4XHX7t2LRITExU+lpmZiSdPnmDkyJFo3rw5SktLcfToUUyePBnx8fFYunRpjde/efNmlJWVYezYsUrLXLx4EWvWrMHff/+N9PR0cBwHV1dXDB8+HB988IHC62rsYmJisGbNGrRv3x7e3t64du2a0rJ//vknQkJC8Oqrr2Lt2rW4efMm/vvf/yI9PR0bNmyQlXNycsKwYcOwfPlyDB06VAtXQRRihGjYwoULGQCWkZGh1fOamZmxSZMmafWcUgkJCQwA++mnnzS6T0Oza9cuBoDt2bNHti09PZ1ZW1uzsWPH1rj/e++9x0xMTNjjx49l244ePcoAsO+//77afRMTExnHceztt99W+HhaWhqzsrJiixcvZgDYsmXLVLqmN954g5mZmbGysrIay3bs2JFNmDBB4WOlpaVsxowZjOM41qtXL7Z8+XL2xx9/sH379rGlS5cyf39/ZmxszNatW6dSvfRJff8HZGVlsby8PMYYY8uWLWMAWEJCgsKy7du3Z35+fqy0tFS27bPPPmMcx7G4uDi5snv37mUcx7EHDx7UqV6k/qg7hmhNZmYmRo8eDUtLS9ja2mL27NkoKiqqUu6XX35B586dYWJiAhsbG4wZMwZJSUlyZe7du4cRI0bAyckJxsbGcHNzw5gxY5CbmwsA4DgOBQUF2Lp1q6wJfvLkyQrrlZaWBgMDA3z55ZdVHouPjwfHcVi3bh0A4NmzZ/j444/h6+sLc3NzWFpaYuDAgbh+/Xo9/zrK/fPPP+jVqxfMzMxgbW2NYcOGIS4uTq7M8+fPMWfOHHh6ekIkEsHBwQGvv/66XDdHTX8zbdi7dy8cHR0RGhoq22Zvb4/Ro0fjt99+Q3FxcbX779u3D2+88QaaN28u2xYUFIS2bdti9+7d1e67c+dOMMYwfvx4hY/PmzcPXl5emDBhQi2uCPD09ERhYSFKSkqqLZeQkIAbN24gKChI4eNTpkzBjh07cPjwYZw8eRIfffQR3njjDYSGhiIiIgJXr17Fxo0b8fHHH2Pjxo1V9k9OTsbUqVPh6OgIkUgEHx8fbN68Wa5MdHQ0OI7Drl27MH/+fDg5OcHMzAxDhw6t8h4DgD179sjei3Z2dpgwYQKSk5OrlLtz5w5Gjx4Ne3t7mJiYwMvLC5999lmVcjk5OZg8eTKsra1hZWWFKVOmoLCwsNq/GwDY2NjAwsKixnK3b9/G7du3MX36dBgYvGzof//998EYw969e+XKS5+L3377rcZjE82g7hiiNaNHj4anpyciIyNx7tw5rFmzBtnZ2di2bZuszFdffYUvvvgCo0ePxltvvYWMjAysXbsWvXv3xtWrV2FtbY2SkhIEBwejuLgYs2bNgpOTE5KTk3Hw4EHk5OTAysoKP//8M9566y1069YN06dPBwC0atVKYb0cHR3Rp08f7N69GwsXLpR7bNeuXRAKhRg1ahQA4OHDhzhw4ABGjRqFFi1aIC0tDd9//z369OmD27dvw8XFRa1/s2PHjmHgwIFo2bIlFi1ahBcvXmDt2rXo2bMnrly5Ak9PTwDAu+++i71792LmzJlo3749srKycPr0acTFxaFTp04q/c2UKSwsVOmDQigUolmzZtWWuXr1Kjp16lRlPES3bt2wadMm3L17F76+vgr3TU5ORnp6Orp06VLlsW7duuHw4cPVnnv79u1wd3dH7969qzx24cIFbN26FadPnwbHcdUe58WLFygoKEB+fj7+/fdf/PTTTwgMDISJiUm1+509exYA0KlTpyqP/fzzz9i/fz/Onz8PHx8fAABjDAUFBTA3NwfAB/Fvvvkm7OzsMGrUKAwcOBAeHh4A+EC6e/fu4DgOM2fOhL29Pf78809MmzYNeXl5mDNnjtz5vvrqK3Ach08//RTp6elYtWoVgoKCcO3aNdl1bNmyBVOmTEHXrl0RGRmJtLQ0rF69GmfOnJG9FwHgxo0b6NWrFwwNDTF9+nR4enriwYMH+OOPP/DVV1/JnXf06NFo0aIFIiMjceXKFfzvf/+Dg4MDvvnmm2r/dqq6evUqAFR5jbi4uMDNzU32uJSVlRVatWqFM2fO4MMPP1RLHUgt6bglhjQB0qbYoUOHym1///33GQB2/fp1xhhjjx49YkKhkH311Vdy5W7evMkMDAxk269evVqlSV+R2nTHfP/99wwAu3nzptz29u3bs9dee012v6ioiInFYrkyCQkJTCQSscWLF8ttgxq6Y/z9/ZmDgwPLysqSbbt+/ToTCARs4sSJsm1WVlZsxowZSo+t6t9MEenzV9PNw8OjxmOZmZmxqVOnVtl+6NAhBoAdOXJE6b4XL15kANi2bduqPDZ37lwGgBUVFSncNzY2lgFgn3zySZXHJBIJ69atm6w7SPo8KOuOiYyMlLvufv36scTERKX1lvr8888ZAPb8+fMq52/RogVbtWqVbNtvv/3GXFxcGADWvHlz9tdff8l1QQwfPpzNnz9fVn7atGnM2dmZZWZmyh17zJgxzMrKihUWFjLGGDtx4gQDwFxdXWXdG4wxtnv3bgaArV69mjHGWElJCXNwcGAdOnRgL168kJU7ePAgA8AWLFgg29a7d29mYWEh10UmvS4p6Wuo8nM/fPhwZmtrW+PfrqLqumOkjyl6Prp27cq6d+9eZXv//v2Zt7d3repA1Ie6Y4jWzJgxQ+7+rFmzAED2DTYqKgoSiQSjR49GZmam7Obk5IQ2bdrgxIkTACD71v7XX3+p9A1dFaGhoTAwMMCuXbtk22JjY3H79m2EhYXJtolEItm3eLFYjKysLJibm8PLy6tWMzxUkZKSgmvXrmHy5MmwsbGRbe/YsSNef/11uW/+1tbWOH/+PJ4+farwWPX5m02cOBFHjx6t8bZ9+/Yaj/XixQuIRKIq242NjWWPV7cvgDrtL62boq6YLVu24ObNmyp/Gx87diyOHj2KHTt2YNy4cTXWWyorKwsGBgaylg2py5cvIz09HdOmTQPAt/iMHTsW3bp1w759+/Dhhx9i6tSpcvuEhIQgOjoaAN9ism/fPgwZMgSMMbn3TnBwMHJzc6u8NidOnCjXvTFy5Eg4OzvLXlOXLl1Ceno63n//fdnfFgAGDx6Mdu3a4dChQwCAjIwMnDx5ElOnTpXrIgOgsEXp3Xfflbvfq1cvZGVlIS8vr8a/nypqeo0oep6aNWumV1OHmxrqjiFaU3nmQ6tWrSAQCGRT7e7duwfGmNIZEoaGhgCAFi1aIDw8HCtXrsT27dvRq1cvDB06FBMmTKi2W6E6dnZ26NevH3bv3o0lS5YA4LtiDAwM5MYvSCQSrF69Gt999x0SEhIgFotlj9na2tbp3Mo8fvwYAODl5VXlMW9vb/z1118oKCiAmZkZvv32W0yaNAnu7u7o3LkzBg0ahIkTJ6Jly5YA6vc3a9mypew49WViYqJw3Id0bFB1XRrSx2q7P2MMO3bsQIcOHdCxY0e5x/Ly8hAREYG5c+fC3d1dpWvw8PCQdYOMHTsW06dPR1BQEOLj42vsklHk8uXL6NKliyw42b59O1xdXbF3714IhUIAfJA5ZcoU2T6Ojo6yackZGRnIycnBpk2bsGnTJoXnSE9Pl7tf+T3GcRxat24tey9W99pr164dTp8+DYDvngSADh06qHStlQMVafdddnY2LC0tVTpGdWp6jSh7fdTUBUc0h1pCiM5UfuNLJBJwHIcjR44o/Kb9/fffy8quWLECN27cwPz58/HixQt88MEH8PHxwZMnT+pcnzFjxuDu3buy6X+7d+9Gv379YGdnJyuzdOlShIeHo3fv3vjll1/w119/4ejRo/Dx8dFp8qnRo0fj4cOHWLt2LVxcXLBs2TL4+Pjgzz//lJWp698sPz8fqampNd4q5+pQxNnZGSkpKVW2S7dVN6bG2dlZrmzl/W1sbBR+Az5z5gweP36ssBVk+fLlKCkpQVhYGB49eoRHjx7J/h7Z2dl49OhRjQNOR44ciaSkJJw8ebLacra2tigrK8Pz58/ltmdlZcld96NHj/DKK6/IAhCAH/NSUVJSkizolb7uJkyYoLSVqmfPntXWTVsqXlNFTMU8KzWp6TWi6PWVnZ0t9x4n2kUtIURr7t27hxYtWsju379/HxKJRDa4slWrVmCMoUWLFmjbtm2Nx/P19YWvry8+//xznD17Fj179sTGjRvx3//+F4Di5uDqhISE4J133pF1ydy9excRERFyZfbu3Yu+ffvixx9/lNuek5Oj9n9k0m/b8fHxVR67c+cO7OzsYGZmJtvm7OyM999/H++//z7S09PRqVMnfPXVVxg4cKCsTE1/M0WWL1+ucOaQovpWl0AKAPz9/XHq1ClIJBK5wannz5+Hqalptc+7q6sr7O3tcenSpSqPXbhwAf7+/gr32759OziOk3WdVJSYmIjs7GzZYNCKli5diqVLl+Lq1atKjw287AKoaZZRu3btAPCzZCq2yFhaWsrt6+TkhAsXLsjtK21xAPgP7B9//FE2s8Pe3h4WFhYQi8VKZ95Udu/ePbn7jDHcv39fVq+Kr73XXntNrmx8fLzscWkLWWxsrErn1TTp83Tp0iW5wO3p06d48uSJbJB6RQkJCfDz89NWFUkl1BJCtGb9+vVy99euXQsAsg/J0NBQCIVCfPnll1W+GTHGkJWVBYBvQq+c/dDX1xcCgUCuGdbMzAw5OTkq18/a2hrBwcHYvXs3fv31VxgZGVVJdCYUCqvUbc+ePQqnLdaXs7Mz/P39sXXrVrnriI2Nxd9//41BgwYB4MemVP4AdHBwgIuLi+zvoerfTBF1jgkZOXIk0tLSEBUVJduWmZmJPXv2YMiQIXItGQ8ePMCDBw/k9h8xYgQOHjwoN530+PHjuHv3rmwGU0WlpaXYs2cP/vOf/1TpCgCADz74APv375e7SVvcJk+ejP3798sCZ2UtPT/++CM4jlM466WiwMBAAKgSRHl7e+PixYuyFo1hw4bh6tWrWLBgAR4+fIhTp05h7ty5APjZHyNGjMCTJ08we/ZsAPxrcsSIEdi3b5/CYEBRvbdt2ybXIrN3716kpKTI3otdunSBg4MDNm7cKPf6+PPPPxEXF4fBgwcD4AOg3r17Y/PmzVWSvKmrdaM2fHx80K5dO2zatEmuq3TDhg3gOA4jR46UK5+bm4sHDx6gR48e2q4qkdLJcFjSpEhHxvv6+rIhQ4aw9evXswkTJjAAbNy4cXJlpTMPevTowb799lu2YcMG9sknn7A2bdrIZivs37+fubq6sjlz5rDvvvuOrVmzhnXt2pUZGhqymJgY2bEGDRrEzMzM2IoVK9jOnTvZuXPnaqzrL7/8wgAwCwsLNmTIkCqPL1iwgAFgkydPZps2bWKzZs1iNjY2rGXLlqxPnz6ycuqaHXP06FFmYGDA2rVrx5YtW8YWL17M7O3tWbNmzdjDhw8ZY4xlZ2fLZgKtXLmSbdq0iY0ePZoBYCtWrKjV30zTysrKWPfu3Zm5uTn78ssv2fr165mPjw+zsLBgd+7ckSvr4eFRZcZNYmIis7W1Za1atWJr1qxhS5cuZc2aNWO+vr4KZ8b88ccfDADbuHGjynVUNjtm9uzZrEuXLuzzzz9nmzZtYl9//TXr2rUrA8BmzZql0rE7dOhQJSlbUVERs7KyYvv375dtW7p0KRMIBAwAMzAwYKtXr5bNxunfv7/suZdKTU1lHh4ezNTUlM2ePZt9//33LDIyko0aNYo1a9ZMVk46O8bX15d17NiR/d///R+bN28eMzY2Zq1bt2YFBQWysj/99BMDwAICAtiqVatYREQEMzU1ZZ6eniw7O1tW7tq1a8zc3JzZ2tqyiIgItmnTJjZ//nzm5+cnK6MsWZn0HMoSj0nl5OSwJUuWsCVLlrABAwYwAOyjjz5iS5YsYWvXrpUr+8cffzCO49hrr73GNm3axD744AMmEAgUJqnbu3cvA8Du379f7fmJ5lAQQjRO+g/o9u3bbOTIkczCwoI1a9aMzZw5U276n9S+ffvYf/7zH2ZmZsbMzMxYu3bt2IwZM1h8fDxjjLGHDx+yqVOnslatWjFjY2NmY2PD+vbty44dOyZ3nDt37rDevXszExMTBkCl6bp5eXmy8r/88kuVx4uKithHH33EnJ2dmYmJCevZsyeLiYlhffr00UgQwhhjx44dYz179mQmJibM0tKSDRkyhN2+fVv2eHFxMZs7dy7z8/NjFhYWzMzMjPn5+bHvvvtOVkbVv5k2PHv2jE2bNo3Z2toyU1NT1qdPH3bx4sUq5RQFIYzx02379+/PTE1NmbW1NRs/fjxLTU1VeK4xY8YwQ0NDuSnONVEWhPz999/sjTfeYC4uLszQ0JBZWFiwnj17sp9++kluOmp1Vq5cyczNzWVTZqUWLlzIWrZsyZ49eybblpyczE6ePCm7ttOnT7P09HSlx05LS2MzZsxg7u7uzNDQkDk5ObF+/fqxTZs2ycpIg5CdO3eyiIgI5uDgwExMTNjgwYOrTLFljM9w+8orrzCRSMRsbGzY+PHj2ZMnT6qUi42NZcOHD2fW1tbM2NiYeXl5sS+++ELu+uoThEifE0U3Ra+R/fv3M39/fyYSiZibmxv7/PPPWUlJSZVyYWFh7D//+U+15yaaxTGmgzYzQghpgnJzc9GyZUt8++23sim5AD9zo2fPnhAKhfjtt99kAywr27t3L4YPH650gGdNoqOj0bdvX+zZs6dK10RTk5qaihYtWuDXX3/FsGHDdF2dJovGhBBCiJZYWVnhk08+wbJly+RmUxkbG+Pw4cPgOA5eXl749NNPcfLkSTx+/Bh37tzBtm3bEBgYiEmTJqk9H01TtWrVKvj6+lIAomPUEkKIhpWUlODZs2fVlrGysqpTjgnSuJSUlGDdunVYt24dEhISZNuNjY0xfPhwfPnllzWuNFwdagkh+oam6BKiYWfPnkXfvn2rLfPTTz8pXWCPNB1GRkYIDw9HeHg4Hj16hOTkZBgbG8Pb2xumpqa6rh4hakctIYRoWHZ2Ni5fvlxtGR8fH6XjAAghpLHSeRCyfv16LFu2DKmpqfDz88PatWurZAeUunXrFhYsWIDLly/j8ePH+L//+78qq0OePHkSy5Ytw+XLl5GSkoL9+/dXyfVACCGEEN3T6cDUXbt2ITw8HAsXLsSVK1fg5+eH4ODgKuscSBUWFqJly5b4+uuv4eTkpLBMQUEB/Pz8qiTGIoQQQoh+0WlLSEBAALp27Yp169YB4NdAcHd3x6xZszBv3rxq9/X09MScOXOqtIRUxHFcnVpCJBIJnj59CgsLC1rYiBBCCKkFxhieP38OFxcXueUZFNHZwNSSkhJcvnxZbm0OgUCAoKAgxMTEaLUuxcXFcqmJk5OT0b59e63WgRBCCGlMkpKS4ObmVm0ZnQUhmZmZEIvFcHR0lNvu6OiIO3fuaLUukZGRChfoSkpKUsvy0oQQQkhTkZeXB3d3d1hYWNRYlqboAoiIiEB4eLjsvvQPaGlpSUEIIYQQUgeqDGfQWRBiZ2cHoVCItLQ0ue1paWlKB51qikgkklu9kxBCCCGap7PZMUZGRujcuTOOHz8u2yaRSHD8+HHZkteEEEIIabx02h0THh6OSZMmoUuXLujWrRtWrVqFgoICTJkyBQAwceJEuLq6IjIyEgA/mPX27duy35OTk3Ht2jWYm5ujdevWAID8/Hzcv39fdo6EhARcu3YNNjY2aN68uZavkBBCCCHK6DxZ2bp162TJyvz9/bFmzRoEBAQAAF599VV4enpiy5YtAIBHjx6hRYsWVY7Rp08fREdHA3i5NkJlkyZNkh2nJnl5ebCyskJubi6NCSGEEEJqoTafoToPQvQRBSGEEEJI3dTmM1SnGVMJIYQQ0nTRFF1CCCGkCRIzhlM5OUgpKYGzkRF6WVtDqOUs4RSEEEIIIU1MVEYGZt+/jycVsoW7iURY3bo1Qu3ttVYP6o4hhBBCmpCojAyMvHVLLgABgOTiYoy8dQtRGRlaqwsFIYQQQkgTIWYMs+/fh6IZKdJtc+7fh1hLc1YoCCGEEEKaiFM5OVVaQCpiAJKKi3EqJ0cr9aEghBBCCGkiUkpK1FquvigIIYQQQpoIZyMjtZarLwpCCCGEkCail7U13EQiKJuIywFwF4nQy9paK/WhIIQQQghpIoQch9WtWyscmCoNTFa1bq21fCEUhBBCCCFNSKi9PVoYG1fZ7iYSYa+Pj1bzhFCyMkIIIaQJuV1QgISiIggB7OvQAYViMWVMJYQQQojmbU9LAwAMsrXFMDs7ndaFumMIIYSQJkLCmCwImeDoqOPaUBBCCCGENBlncnPxuLgYFkIhhtja6ro6FIQQQgghTYW0FWSEvT1MhEId14aCEEIIIaRJKJZIsLt8cTp96IoBKAghhBBCmoQ/s7KQXVYGFyMjvKqlZGQ1oSCEEEIIaQK2p6cDAMY5Omp9Kq4yFIQQQgghjVxOaSn+yMwEAIx3cNBxbV6iIIQQQghp5PZlZqKYMfiYmsLP3FzX1ZGhIIQQQghp5CrmBuH0pCsGoCCEEEIIadSSiooQnZMDABirJ7NipCgIIYQQQhqxnenpYAB6W1nBQ8HCdbpEQQghhBDSiP2iR2naK9OLIGT9+vXw9PSEsbExAgICcOHCBaVlb926hREjRsDT0xMcx2HVqlX1PiYhhBDSGN3Iz8fNggIYcRxG2tvrujpV6DwI2bVrF8LDw7Fw4UJcuXIFfn5+CA4ORnr5fObKCgsL0bJlS3z99ddwcnJSyzEJIYSQxkg6IHWwrS2aGRrquDZVcYwxpssKBAQEoGvXrli3bh0AQCKRwN3dHbNmzcK8efOq3dfT0xNz5szBnDlz1HZMAMjLy4OVlRVyc3NhaWlZtwsjhBDSpIgZw6mcHKSUlMDZyAi9rK11mhRMwhg8zp3Dk+Ji7PPxQaiWWkJq8xlqoJUaKVFSUoLLly8jIiJCtk0gECAoKAgxMTFaO2ZxcTGKi4tl9/Py8up0bkIIIU1TVEYGZt+/jycVPkvcRCKsbt1aax/+lZ3MycGT4mJYCYUYZGOjkzrURKfdMZmZmRCLxXCsNFjG0dERqampWjtmZGQkrKysZDd3d/c6nZsQQkjTE5WRgZG3bskFIACQXFyMkbduIap80Thtkw5IHeXgAGM9WDFXEZ2PCdEHERERyM3Nld2SkpJ0XSVCCCENgJgxzL5/H4rGNUi3zbl/H2Itj3woEouxR89WzFVEp90xdnZ2EAqFSCuP1qTS0tKUDjrVxDFFIhFEIlGdzkcIIaTpOlXe5aEMA5BUXIxTOTl4tVkzrdXr0LNnyBOL4S4SoZeVldbOW1s6bQkxMjJC586dcfz4cdk2iUSC48ePIzAwUG+OSQghhCiSUlKi1nLqIu2KGefgAIEepWmvTKctIQAQHh6OSZMmoUuXLujWrRtWrVqFgoICTJkyBQAwceJEuLq6IjIyEgA/8PT27duy35OTk3Ht2jWYm5ujdevWKh2TEEIIUQdnIyO1llOHZ6WlOJSVBUC/u2IAPQhCwsLCkJGRgQULFiA1NRX+/v44cuSIbGBpYmIiBIKXDTZPnz7FK6+8Iru/fPlyLF++HH369EF0dLRKxySEEELUoZe1NdxEIqVdMhz4WTK9rK21Vqe9GRkoZQx+ZmbooEcr5iqi8zwh+kjdeUL0be44IYQQ9fk5NRUT79xR+BgHYK8Wc3QAQO+rV3EqNxfftmyJuc2ba+28Ug0mT0hToI9zxwkhhKhPVmkpAMCA41BW6Xv9LFdXrf6vf/TiBU7l5oIDMNbBQWvnrSuaoqtB+jp3nBBCiHqIGcOa5GQAwNrWrXHCzw87vL0xrXw25o70dFmQog07ypcn6WttDTc9WzFXEQpCNERf544TQghRn4NZWUgoKoKNgQEmOjnh1WbNMNbREd+1bQsfU1NklpbikwcPtFIXxphsVsz4BjIGkoIQDanN3HFCCCEN06onTwAA011cYFohK6mRQIDvvbwAAJtTU/GvFv7XX8vPR1xhIUQchxENpLufghAN0de544QQQtTj2vPniM7JgRDA+y4uVR7vaWWFd5ydAQDvxMejWCLRaH2krSBD7exgZdAwhnxSEKIh+jh3nBBCiPpIx4KMtLeHu5LxF5EtW8LR0BDxL17gm8REjdVFzBh2lo8HaShdMQAFIRojnTte3URcB0NDrc4dJ4QQoh7pJSXYXt7yMMfNTWm5ZoaGWFWeSPOrx49xt7BQI/U5kZ2NlJIS2BgYYKCerpirCAUhGiLkOKwuf+EpC0TyxGLE5OZqr1KEEELUYuPTpyhhDN0sLNC9hrVZwhwcMMDGBiWM4d27d6GJ9FzSrpjRDg4wEjScj/aGU9MGKNTeHnt9fOBaaXE8N5EIvqamKJJIMODGDZykwamEENJgFEsk2PD0KYDqW0GkOI7Dd23awEQgwImcHGyrtMBqfRWKxYjKzAQAjG8AuUEqoiBEw0Lt7fGoe3fZ3PETfn541L07znfujNebNUOBRIKBN24gOjtb11UlhBCigt3p6UgtKYGLkRFGqjgLpYWJCRZ5egIAPrp/H5lqnJTwR1YWnovF8DQ2Rg89XjFXEQpCtEDIcbK54682awYhx8FEKMRvHToguFkzFEokGHTzJv6hQIQQQvQaY0w2LXeGqysMa9H18aGbGzqamSGrrAwfqzF3iCw3iJ6vmKsIBSE6ZCIU4kCHDhhkY4MXEgkG37yJY8+e6bpahBBClDiTm4sr+fkwFggwvXz6raoMBQJ837YtOABb09LU8sUzs6QER8o/NxrSrBgpCkJ0zFgoRFSHDhhsY4MiiQRDYmPxNwUihBCil6StIBMcHWFXhxQL3a2s8F55TpF3795FkVhcr/rszshAGWPoZG4ObzOzeh1LFygI0QMigQD7OnTAEFtbFEkkGHrzJo5kZem6WoQQQip4XFSE/eUDQGe7utb5OEtbtoSzkRHuvXiByHrmDpF2xUxogK0gAAUhekMkEGCvjw9C7OxQzBiGxcbiMAUihBCiN9YlJ0MCIKhZM3QwN6/zcawMDLCmPIVDZGIi4goK6nScBy9eICYvDwIAYxrYrBgpCkL0iJFAgN3t2yPUzg4ljGF4bCwOlkfdhBBCdCe/rAw/lE/LrU8riNQIe3sMtrFBKWN45+5dSOqQO2RHeStIv2bN4FwpFURDQUGInjEUCPBr+/YYZW+PEsYQeusWfqNAhBBCdGpbWhpyxWK0NjHBIFvbeh+P4zisb9sWpgIBTuXm4qfU1FrtX3HF3IbaFQNQEKKXDAUC7PD2Rpi9PUoZw8hbt7A/I0PX1SKEkCZJwhhWlw9Ine3qqrZpsB7GxljcogUAYO6DB0ivRe6QS8+f4+6LFzARCDDczk4t9dEFCkL0lIFAgF+8vTHWwQFljGH07dvYR4EIIYRo3V/PnuHuixewFAoxyclJrcee7eoKf3NzZJeVIfz+fZX3k65bM8zODhYNZMVcRSgI0WMGAgG2tWuH8eWBSNitW9hTvkoiIYQQ7ZBOy33L2VntH/gGAgE2tW0LAYDt6ek4qkKKhjKJRLZibkPuigEoCNF7BgIBtnp7Y6KjI8QAxt6+jV0UiBBCiFbcLijA39nZEACYqYYBqYp0tbSUHfu9u3fxoobcIceys5FeWgo7Q0P0b9ZMI3XSFgpCGgAhx2Fzu3aY7OQEMYBxt2/LRkWLGUN0djZ2pqUhOjsbYg2szkgIIU3VmvJWkGF2dmhhYqKx8yxp0QKuRkZ4UFSE/z5+XG3Z7eVfRMPs7WuVNl4fNdyOpCZGyHH40csLQgA/pqbizbg4nM/LQ1RmJp4UF8vKuYlEWN26NUJVXFSJEEKIYlmlpbIVb1VZLbc+LA0MsLZNG4TeuoVvk5IwztERPgoyoOaXlSGqfHxgQ++KAaglpEERcBw2eXnhbWdnSACsSU6WC0AAILm4GCNv3ZK9SAkhhNTND0+f4oVEAn9zc/TSwuq0w+3tMczWFmWMYXp8vMLcIb9lZaFQIkErY2MEWFpqvE6aphdByPr16+Hp6QljY2MEBATgwoUL1Zbfs2cP2rVrB2NjY/j6+uLw4cNyj6elpWHy5MlwcXGBqakpBgwYgHv37mnyErRGwHFY36YNzJQ0wUlfsnPu36euGUIIqaNSiQTry5OTzXFzA6el1WnXtmkDc6EQZ/Py8L+UlCqPS2fFjHd01FqdNEnnQciuXbsQHh6OhQsX4sqVK/Dz80NwcDDSlQy+PHv2LMaOHYtp06bh6tWrCAkJQUhICGJjYwHwCVxCQkLw8OFD/Pbbb7h69So8PDwQFBSEgjqmxtU3Z3JzUSCRKH2cAUgqLsapnByt1YkQQhoTaVe3g6GhVlOiuxsb47/luUM+efAAqRVau9NKSmQLnDbEFXMV0XkQsnLlSrz99tuYMmUK2rdvj40bN8LU1BSbN29WWH716tUYMGAA5s6dC29vbyxZsgSdOnXCunXrAAD37t3DuXPnsGHDBnTt2hVeXl7YsGEDXrx4gZ07d2rz0jQmRcWENqqWI4QQIk+anOw9FxeItDz4c6arKzqbmyNXLMaHDx7Itu9KT4cYQDcLC7Q1NdVqnTRFp0FISUkJLl++jKCgINk2gUCAoKAgxMTEKNwnJiZGrjwABAcHy8oXl0eNxsbGcscUiUQ4ffq0wmMWFxcjLy9P7qbPnFVcPlrVcoQQQl46n5eHmLw8GHEc3nVx0fr5heXj/wQAfk1Px6HMTERnZ8tm6oxroIvVKaLTICQzMxNisRiOlZqVHB0dkaokj35qamq15du1a4fmzZsjIiIC2dnZKCkpwTfffIMnT54gRUH/GgBERkbCyspKdnN3d1fD1WlOL2truIlEqK43UAiglMaEEEJIrUlbQcY4OMBJRwvDdbKwwOzyGTnDYmPR9/p1PCgqAgB8k5TUaCYf6Lw7Rt0MDQ0RFRWFu3fvwsbGBqampjhx4gQGDhwIgZImtYiICOTm5spuSUlJWq517Qg5DqvLl4FWFoiIAfS/cQNvx8cjp7RUa3UjhJCGLLm4GHvKP+Bna3habk06m5sD4P+fV5RaUtJoZkHqNAixs7ODUChEWvloX6m0tDQ4KcnP7+TkVGP5zp0749q1a8jJyUFKSgqOHDmCrKwstGzZUuExRSIRLC0t5W76LtTeHnt9fOBaKUp3F4nwc7t2eL+8CfF/KSlof/EiDjSCFyshtUXJ/EhtfZecjDLG0NvKCp0sLHRWDzFjmJeQoPCxxjQLUqdBiJGRETp37ozjx4/LtkkkEhw/fhyBgYEK9wkMDJQrDwBHjx5VWN7Kygr29va4d+8eLl26hGHDhqn3AnQs1N4ej7p3xwk/P+zw9sYJPz8kdO+OCU5OWN+2LU76+6OtiQlSSkow/NYtjLp1S26kNSGNWVRGBjzPnUPf69cxLi4Ofa9fh+e5c43i2yPRjBdiMb4vn5ar61aQUzk5VfJAVdRYZkHqvDsmPDwcP/zwA7Zu3Yq4uDi89957KCgowJQpUwAAEydOREREhKz87NmzceTIEaxYsQJ37tzBokWLcOnSJcycOVNWZs+ePYiOjpZN03399dcREhKC/v37a/36NE3IcXi1WTOMdXTEq82aQVhh3ngva2tc79IFEc2bQwhgb0YG2l+8iC0pKWANPHompDpRGRkYeesWJfMjtbI9LQ1ZZWXwNDbGMDs7ndalqcyC1Hna9rCwMGRkZGDBggVITU2Fv78/jhw5Iht8mpiYKDeWo0ePHtixYwc+//xzzJ8/H23atMGBAwfQoUMHWZmUlBSEh4cjLS0Nzs7OmDhxIr744gutX5s+MBYKsbRlS4y2t8e0+Hhcyc/HlPh47EhPx6a2beGpwbUQCNEFMWOYff8+FIXZDPw4qjn372OYnZ1c0E6aNsaYbLXcWa6uOn9tNJVZkByjr8RV5OXlwcrKCrm5uQ1ifIiqyiQSrHzyBAsfPUKRRAJTgQBLW7bETD14wxGiLtHZ2eh7/XqN5U74+eHVBr4CKVGf49nZCLp+HWYCAZ4EBsLa0FCn9REzBs9z55BcXKwwoObArxWW0L273v3/rs1nqM67Y4j2GAgE+KR5c9zo0gV9rKxQKJFgzv37+M/Vq7jdSLLJEtJUmrGJeklbQaY4O+s8AAGqnwUpvb+qdWu9C0Bqi4KQJqiNqSn+8ffHxrZtYSkU4lxeHvwvXcLiR49QUp4OnmYVkIaqqTRjE/W5V1iIg1lZAPiuGH2hbBakm0iEvT4+jWK1dJ2PCSG6IeA4vOPigsE2Nnj/3j38kZWFhY8eYU9GBiY4OGDd06dyg/rcRCKsbt26UbzoSeMmTeanbGaBtBm7l7W1VutF9Nfa5GQAwGAbG71Lhx5qb49hdnY4lZODlJISOBsZoZe1dYNvAZGiMSEKNNYxIcowxrA7IwOz7t1DhpLEZtKXe2OJvknjtjc9HaNu31b4GAd6HZOXckpL4RYTgwKJBEc7dkSQjY2uq9Tg0ZgQUiscxyHMwQE3u3SBqZKsso0pOQ5p/EyFQgCKMwp3t7SkAITIbE5NRYFEAh9TU/SjgcpaR0EIkYkrLERh+ZgQRRpLchzS+C0vX3phjqurLJnfD23bQgAgJi8PBzMzdVtBohfEjMm6Yma7uYFrJF0cDQmNCdECsRg4dQpISQGcnYFevYDyL2p6RdXZAg+LivCqZqtCSJ1dfv4cJ3JyYMBx+NDdHe4VVtSOf/ECy5OSMOPePbxqbQ1zA/oX2JT9npmJR0VFsDEwwPhKC6MS7aCWEA2LigI8PYG+fYFx4/ifnp78dn2j6myBD+7dw/yHD5FGUxyJHlpR3goSZm8vF4AAwCJPT3iIREgsLsbCR490UDuiaxVn/i0qfw284+Ii68Ij2kVBiAZFRQEjRwLl089lkpP57foWiEhnFVTXIGkAoEAiQWRiIjxiYvBOfDzuFRZqq4qEVOtxURF2p6cDAD5yd6/yuJlQiA1t2wLg80Jcef5cq/UjulV5PaEb5fmRWlQKVon2UBCiIWIxMHs2oGgMp3TbnDl8OX1RU3IcDsDO9u2x38cH3S0tUcwYNqWkwOvCBYyMjcXFvDxtV5kQOaufPIEYQD9ra7yiZAXUgba2CLO3hwTA9Ph4lFUzDoo0HsrWEwKAd+7epfWEdISCEA05dapqC0hFjAFJSXw5fVJTcpyRDg4IsbfH2VdewUl/fwy2sQEDsC8zE92uXEHfa9dwJCuLFsgjWpdTWoofUlIAAB8raAWpaFXr1rA2MMDl/HysKx+YSBqv6tYTkqKZf7pBo7I0pPx/odrKaZMqyXE4jkMva2v0srZGbH4+liUlYUd6OqJzchCdk4OOZmb4pHlzjLa3h2GFab9ixhpt0h2iW5tSUpAvFqODmRmCa8j14CQS4ZuWLfHO3bv4PCEBofb2aE5N8o3WqZwcpcnrAPmZf7SekHZRS4iGODurt5y2CTkOrzZrhrGOjni1WbNqA4UO5ubY6u2NhwEB+NDNDWYCAW4UFGBCXBzanD+PNU+eoEAsrtIf2/f6dXieO0fNoKTeSiQSrC5vevzY3V2lqZZvOTujp6UlCiQSzLx3j1rvGjFaT0h/URCiIb16AW5ugLL/hRwHuLvz5RoLd2NjrGzdGkmBgfhvixawNzTE4+JizL5/H05nzmCEgv7Y5OJijLx1iwIRUi8709PxtKQELkZGGOvgoNI+Ao7D915eMOQ4/JGVhf2UO6TRovWE9BcFIRoiFAKrV/O/KwtEVq3Sz3wh9dXM0BCfeXjgcffu2NCmDVqKRMhXMviPMrGS+mKMyZKTfeDmBiMlWX8V8TEzwyfl40dm3buH3LIyjdSR6JaJQIDq/tVyANxpPSGdoCBEg0JDgb17AUWLMn71Ff94Y2YiFOJdV1dsateu2nKUiZXUx9/Z2YgtKIC5UIh36tC/+ZmHB1qbmOBpSQk+e/hQAzUkurQtNRV9rl2DdCKiopl/AD9YmcanaR8FIRoWGgo8egScOAHs2AEEB/Pb79zRabW0Kp36Y4kGSVtB3nJ2hrWhYa33NxEKsbE8d8h3T5/iXG6uWutHdKNUIsEH9+5h0p07KGYMQ2xtsa1dO6Uz/2g9Id2g2TFaIBQCr77K/96yJfDXX8CBA8CLF4CJiS5rph2q9rNaUwptUkvXnj/HsexsCAHMVtTkqKJ+zZphoqMjtqWlYfrdu7jcubPcrC7SsKSXlGDUrVs4WR5QLvTwwAJPTwg4DuMcHWmGnh6hd5mWdesGbNoE3LvXNAIQQLVMrADw/t27OPbsmVbqVBsV0zxHZ2fT2BU9sqJ8RswoBwd41vMNtaJVK9gaGOBmQQFWVpfkh8jRt/fHpbw8dL58GSdzc2EhFOJAhw5Y1KIFBOWBRm1m/hHN4xjNS6siLy8PVlZWyM3NhaWlpa6r0yhIsxUCkEsYxJXftzUwQFb5oMApTk5Y3qoVbOrQtK5uURkZmH3/vtysHjeRCKtbt6bmWx1LKipCy/PnUcYYLnXujM5KMqTWxtbUVEy+cwcmAgFiu3ZFy6byTaGO9O39sTU1Fe/Ex6OYMXiZmOBAhw5oZ2am9Xo0dbX5DKWWEB1rKsMgqsvEus/HBwndu2OWqys4AD+lpqL9hQvYk56u09wNytI8q2tasb59g2xo1iQno4wx9LW2VksAAgATHR3R19oaLyQSvH/3LuUOqYam3x+1IR3/MbnC+I/znTtTANIAUEuIAtpoCblwAZg7F3ByAnbt0sgp9FJNGVPP5ubirfh4xJUvijfU1hbftW1bJXjRRj09z51TmmWRAx9AJXTvXqfmXH37BtnQ5JaVwT0mBs/FYhzy9cUgW1u1HftuYSE6XryIYsaww9sbY2mJ9yo0/f6ojerGfxDdoJaQBkAkAk6e5FfSTUvTdW20p6b+2B5WVrjapQsWenjAkOPwe1YW2l+4gI3JyZBoMV5WNc3zhuRkPHzxAgW1WIlQn75BNlQ/PH2K52Ix2puaYkANKdprq62pKT7z8ADA5695Vlqq1uM3BrVJg65JFyuN//it0vgPov+oJUQBbY0J6dEDiIkBli4FIiI0dpoG61ZBAd6Kj8e58tV5e1tZYZOXF7xMTTV2zhKJBNE5OViemIijtfwHai4UwsnISHZzNDSUu+9kZAQ7Q0P85+pVJCvph9PmN8iGqkQiQctz55BcUoIfvbwwVQNrH5RIJPC/dAlxhYV4y9kZP3h5qf0cDdnOtDSMi4ursdxcd3csadECIg3MNNqSkoJ3796l8R96qDafoRSEKKCtIGTbNmDSJMDTE7h/v3FmT60vMWNYn5yM+Q8fokAigYjjsMDTE3Pd3dU2hTKntBR/PnuG3zIz8eezZ8irRauGk6EhcsVivFDzcvAn/PxoIS0lfklNxZt37sDR0BCPAwM18gEHAKdzctDr2jUAwEl/f8qmWcGfmZkYFBurUllLoRBv2NpihL09BtjYwLSe/+hKJRKEP3ggW/14qK0ttnl7w4qm+OuNBtcds379enh6esLY2BgBAQG4cOFCteX37NmDdu3awdjYGL6+vjh8+LDc4/n5+Zg5cybc3NxgYmKC9u3bY+PGjZq8hDoZNQpo1oxPZvb337qujX4Schw+cHPDrW7dMMDGBsWM4bOEBHS5fBmXyltIgNoP8kwsKsLaJ08QdO0a7M+exbi4OOzKyECeWAwnIyNMc3KCnaGh0mnF0jTPT3r0QEGvXsj9z39wt1s3nPT3x+727bGmdWt81rw5pjk5YbCNDTqbm8PVyEjlNxwlblOscop2TQUgAPAfa2u8Xd7KMv3uXRSrOdBsqBKLijBPhcyy5gIBnAwNkScWY0d6OkbcugW7M2cwIjYWO9LSakyRr+g9nVZSgn7Xr8sCkEWentjfoQMFIA2Yzp+5Xbt2ITw8HBs3bkRAQABWrVqF4OBgxMfHw0HBQlRnz57F2LFjERkZiTfeeAM7duxASEgIrly5gg4dOgAAwsPD8c8//+CXX36Bp6cn/v77b7z//vtwcXHB0KFDtX2JSpmYAJMnA//3f8DGjcDAgbqukf7yMDbGYV9f7EhPx+x793CjoAABV67gQzc3dLKwwKcPH1Y7yJMxhmv5+fgtMxO/ZWXhWn6+3PHbm5pimJ0dhtnZoauFBQQch0HlYzek04ilFKV5tjQwgKWBAdrU0FX0T3Y2+l2/XuP10kJaih3Pzsb1ggKYCgR418VF4+f7pmVL/J6ZiTuFhfg2MRFfeHpq/Jz67GxuLobHxiK9tBSWQiHyxGKl74+t3t4IsbPDubw8RGVkYF9mJh4VFSEqMxNRmZkw4jgENWuGEfb2GGZnB9sKU/IVDdx2MDSEmDFklZXBQijEL97eGGpnp5XrJpqj8+6YgIAAdO3aFevWrQMASCQSuLu7Y9asWZg3b16V8mFhYSgoKMDBgwdl27p37w5/f39Za0eHDh0QFhaGL774Qlamc+fOGDhwIP773//WWCdt5gmJjwfatQMEAr5FpHwtLVKNjJISfHj/PranpystI/1H+IWHB56VleH3zEwkVviHJgDQ08qKDzxsbdFaSfCg6J+hu0iEVXWcxSKdVZBcXAxlbzwRxyG2a1eldWrKBly/jr+yszHL1RVr2rTRyjl/TUvD2Lg4iDgON7p2Rdsm+rxsS03F2/HxKGEMfmZm+N3XF5eeP1f5/cEYw9X8fFlAcqd8BhwACAH0sbbGCHt7iDgOb9+9q/T94WpkhGN+fjT+Q481mDEhJSUlMDU1xd69exESEiLbPmnSJOTk5OC3336rsk/z5s0RHh6OOXPmyLYtXLgQBw4cwPXyb5jTp0/H1atXceDAAbi4uCA6OhpDhw7FoUOH0Lt37yrHLC4uRnGFN1FeXh7c3d21lqzso4+ALl34dWa0PBO1QfsjMxPDY2Oh6ggOU4EA/W1sMMzWFoNtbWGvYmtDTdOKa0tZ4raKzAQCLG/VCu+4uICjAaoAgBv5+fC7dAkCAPcDAtBCS4nEGGMYeOMG/srORl9raxz382tSz4mYMcx/+BDflneDDbezw7Z27WBe3gVS1/dHXEEB9pUHJJVbJqvjamSEx4GBNHBbj9UmCNFpd0xmZibEYjEcK83Dd3R0xB0lK7ylpqYqLJ+amiq7v3btWkyfPh1ubm4wMDCAQCDADz/8oDAAAYDIyEh8+eWX9byauluxQmenbtAshEKVApBBNjZ4z8UF/Zo1g0kdBsVJpxWrizRxm6JvkPOaN8ev6ek4lZuL9+7dw77MTPzPywsexsZqO39DtbL8Q3Ckvb3WAhAA4DgOG9q2hc/FiziRk8NnVdXAjBx99LysDOPi4nAwKwsA8LmHB76slIOjru8PbzMzfG5mhs89PfHwxQtEZWRgc2qqLEeQMsklJTiVk0MDtxsJvRiYqm5r167FuXPn8Pvvv+Py5ctYsWIFZsyYgWPHjiksHxERgdzcXNktqfyfHdFvqg7enODoiDfs7OoUgGhKqL09HnXvjhN+ftjh7Y0Tfn5I6N4d77u6ItrfH6tat4aJQIBj2dnwvXgR/3v6tEln70wuLsaO8u63j3XQZ9nCxASLyseDfPTgATKawMDhhBcv0OPqVRzMyoKI47DD2xtLNJSDo6WJCT5u3hxflOdnqQkN3G48dNoSYmdnB6FQiLRK2brS0tLg5OSkcB8nJ6dqy7948QLz58/H/v37MXjwYABAx44dce3aNSxfvhxBQUFVjikSiSDScT9Idjbwww/8VN1Nm3RalQZD1cGb+jrIU9k3SAHHYbabGwba2GDynTuIycvD23fvYl9mJn5o2xZuTbBVZM2TJyhlDL2trNBVR+s5fejmhu1pabhRUIAP79/HW87OjXYl1pM5OQiNjUVWWRmcjYxwoEMHdNPC372hv6dJ7em0JcTIyAidO3fG8ePHZdskEgmOHz+OwMBAhfsEBgbKlQeAo0ePysqXlpaitLQUgkpT94RCISR6PMUuJweYN48PRB480HVtGoaaVueVTqNtqPkd2pqa4tQrr2BZy5YQcRyOPHuGDhcvYmtqapNqFckrK8PGp08B6KYVRMpQIJAlLdueno6+169jXFwc+l6/Ds9z5xpNptsfU1IQdP06ssrK0NncHBc7d9ZKAAI0/vc0qUrn3THh4eH44YcfsHXrVsTFxeG9995DQUEBpkyZAgCYOHEiIiqkE509ezaOHDmCFStW4M6dO1i0aBEuXbqEmTNnAgAsLS3Rp08fzJ07F9HR0UhISMCWLVuwbds2DB8+XCfXqIoWLYABA/jfqSVENUKOw+rWrQGgyj8tRdNoGyIhx+Hj5s1xtUsXdLOwQK5YjMl37mBobCyeVpM2uzH5MSUFeWIxvExMMFiNa8TUhbJU5Y0h5X6ZRMK38MTHo5QxjLa3x8lXXtHquk1N4T1N5Ok8CAkLC8Py5cuxYMEC+Pv749q1azhy5Ihs8GliYiJSUlJk5Xv06IEdO3Zg06ZN8PPzw969e3HgwAFZjhAA+PXXX9G1a1eMHz8e7du3x9dff42vvvoK7777rtavrzak1du8GWginy/1Vt3qvHt9fBrNYnDeZmY488oriGzRAkYch4NZWehw8SK2p6U16laRUokEq548AQB85O6u0zVBxIxh9v37Ch+TPgNz7t9vkKsh55aVYUhsrOxv/aWnJ35t377e2U3roqm8pwlP53lC9JE284RUVFbGt4g8eQLs2AGMHau1Uzd46p5Gq89i8/Mx6c4dXCmf1hhiZ4eNbdvCsbyfXFN/C138jaVrlDgYGuJx9+4w1uHg4ujsbPRVIdFcQ0u5f7+wEENiY3GnsBAmAgG2tWuHkQoSRWpbU3pPNzYNZooukWdgALz9NrBwIZ9BlYIQ1al7Gq0+62BujnOdOuGbxEQsfvwYBzIzcSonB+vbtoUhx1WZ+ls5e2xdKErapo7jVqdiivaZrq46DUAA1WdkPNXTmRuKPtT/zcnByFu3kF1WBlcjI/zu64tOFha6riqApvWebsqoJUQBXbWEAEByMuDhAYjFwK1bQPv2Wj09aWCu5+djUlwcrhcUKC0j/e5Y16ZsaXK1yv8o6nvcmpzIzsZr16/DRCBAUmCgXFpvXVC1JaSNsTG+adUKw+zs9GZJeUVBpLWBAfLKyiABEGBhgf0dOsCZsiUSNaCWkAbM1ZVf2E4o5FtGCKmOn7k5LnTujCWPHuG/iYkKyzDwAcOc+/cxzM5O1qTNGEMZYyiWSFDMGIokEv536Y0xFIjFeEdJCm1lx1UXaSvIVCcnnQcgwMuZG9Wl3OcA3CsqQuitW/A1M8PnHh4YYW+v024EZUFkTvkCcr0sLfG3n5/OW5pI00QtIQrosiUEABgD9OQLFGkgVP2W3kwoBOM4WcChrje/WsZBXLsGREQAkZG41aYNOly8yH+oBwSglRYzpFZHWcp96dt1s5cX7r14gbXJyXgu5vP5epua4jMPD4TZ28NAg6v+KiJdq0jZrB6An/Ka0L07jbcgalObz1Cdz44hVdH/AlJbqo5XyBaLkVNWhiIFAYgQ/Po6zQwM4GxkBE9jYzir2AJxOje3dhVWZN8+4MgRICoKK8pbQULt7PQmAAFqnrkx2dkZX7Vsicfdu2OhhwesDQwQV1iICXFx8L54EVtSUlCqxXxFp3Jyqg1AACCpuBincnK0UyFCKqGWEAV03RIidesWcOgQ8MknOqsCaSBUbQn5oW1b9LK2hojjIBIIIBIIYFz+U9E3YVWPCwCBlpaY7eaGUDs7GNblG7+/P3D9Okr9/GC2ejVKGUPMK6+gu5VV7Y+lYarO3MgtK8O65GSsTErCs/LuD09jY8xv3hyTnJxgpIGWkQKxGOfz8nAmNxd7MjJws5rxQlI7vL0xttKaXITUVYNZRVdf6UMQkpMDODoCJSXApUtA5846qQZpIKTN7srGK3Dgv63Xttm9puMCfOtJiUSCsvL7biIR3ndxwXQXF9XHcqSlARWWanCIikJbDw+c7tRJ5brqs/yyMmx4+hTLkpKQUVoKgO8G+bR5c0xzcpKNx6jLtNSnxcU4k5vL3/LycPX5c5VXlpZqaNOKiX6jIKSe9CEIAYBx44CdO/lpu5RFldSkpvEK9Z0dU91xe1haYsPTp9j49CnSyz9kjQUCTHB0xGxXV3QwN6/+JNu2AZMmye6+GRGBER9+iJBGlpiqUCzGpqdP8W1SkqwLzdnICJ+4u8PRyAifPHxY7TRoCWO4XVCAM3l5OF0eeCQUFVU5j5tIhP9YWSHQwgJfJSYio7RUrcEpIdWhIKSe9CUIOXkS6NMHMDPjp+7qYas00TOKpmK6i0RYpYE8IYqOWyyR4Nf0dKx+8gRXy5OpAUA/a2vMdnPDYFtbhdNWJWFhwL59EIjFKBUK8Vffvhj499+N9oOxSCzGj6mp+DoxsdoxGxz4wG+cgwNyyspwNi9PNqulYpmOZmboaWWF/1hZoaeVFZpXWORQU8EpIcpQEFJP+hKEMAb4+ABxccD69cD77+usKqQB0YeMqYwxnM7NxeonT7A/MxNOGRlwzM6Gu0iEMAcHDLW1hXn5HPR/nj1D12HDYFFYKNs/z8wMlw4cwGs2Noor4+jIz2dv4IolEvyUkoKZ9+6p3IViKhCgu6UlepYHHN0tLWFVw3x+TQWnhChCQUg96UsQAgBr1gCzZwO+vsD16zRzhjQ8j4uKUPLaa2gTE6O0jITjIKjwr6jy/Sr69QOOHVNnNXVG1cG/M1xcMNnJCX7m5nUa+Etp0Im20BTdRuTNNwETE+DmTaCa/+GE6C0PY2O0+fBDMGtrpYNbKwcc1QYg1tbAO++orX66pur06p5WVuhiaVm3mUd4mQZ9rKMjXm3WjAIQohcoCNFzzZoBY8YA9vb8wnaENEijRoGLjwc3fDgAgNX2A1BafvhwID6eTyvcSDiXLzyornKENCQUhDQA334LJCUBo0fruiaE1IODAxAVBezaBc7KChIV04RLhEJ+VPauXfz+erDCqzpJ08ErC8s48OM3ellba7FWhGgHBSENgJ0dQOtKkUZj9GggPh7ZffvWmDaeAcju25dv/WikUbiQ47C6dWsAqBKISO+vat2auk9Io0RBSAMikQDHj/M/CWnQHBzQLCAA4hpaQ8RCIZp1797oWj8qqykdPM1gIY0VzY5RQJ9mx0gxBnTqxK/xdewYPzmAkAbN3x/s+nWl3RBA+Uq9/v7A1ataqpRu0QwW0hjQ7JhGiOOAHj343zdu1G1dCKm31FSgUgAiKf+wlVT40OUAPvJOS9Nm7XSGZrCQpoaCkAZEOivxwAEgJUWnVSGkfv76S+4uEwohtrTEzfBwiC0twSp301QqTwhpHCgIaUA6duRbQ8rKgM2bdV0bQurh8GFAIJBNveWGDoXh3bvwXbEChnfvghs6lC/HcXy5w4d1WFlCiKZQENLAvPsu/3PTJkBc26UyCdEHZWXAkSP8CGtFU28rTOWFlRVf7s8/6QVPSCNEQUgDM3Ikn8AsMZH/P05Ig/PiBdCy5cvEY8qm3pZP5cXw4UCrVkCFtWUIIY0DBSENjIkJMHky/zu1UJMGycICuHRJtcRj0laRixf5/QghjUr1Sy8SvTR7Nt8iEhio65oQUkcqZkutc3lCSINAQUgD5OHB3wghhJCGTC+6Y9avXw9PT08YGxsjICAAFy5cqLb8nj170K5dOxgbG8PX1xeHK/VLcByn8LZs2TJNXoZO5Obyyct27gSio2nsHiGEkIZD50HIrl27EB4ejoULF+LKlSvw8/NDcHAw0tPTFZY/e/Ysxo4di2nTpuHq1asICQlBSEgIYmNjZWVSUlLkbps3bwbHcRgxYoS2LksrwsL4Qaqvvw6MGwf07Qt4evJd6IQQQoi+03na9oCAAHTt2hXr1q0DAEgkEri7u2PWrFmYN29elfJhYWEoKCjAwYMHZdu6d+8Of39/bFSSSjQkJATPnz/H8ePHVaqTPqZtrywqClAUU0kTLO7dC4SGardOhBBCSINJ215SUoLLly8jKChItk0gECAoKAgxMTEK94mJiZErDwDBwcFKy6elpeHQoUOYNm2a0noUFxcjLy9P7qbPxGJ+cKoi0pByzpz6dc2IxXz3DnXzEEII0RSdBiGZmZkQi8VwdHSU2+7o6IjU1FSF+6Smptaq/NatW2FhYYHQapoFIiMjYWVlJbu5u7vX8kq069Qp4MkT5Y8zBiQl8eVSU4G//wbu3gWKilQ7flQU363Tty918xBCCNGcOgUhW7duxaFDh2T3P/nkE1hbW6NHjx54/Pix2iqnDps3b8b48eNhbGystExERARyc3Nlt6SkJC3WsPZUXTcmJQU4cQIIDga8vPgcI87O/NTesWOBefOACkNpwBgfaIwcWTXISU7mt1MgQgghRF3qFIQsXboUJiYmAPjukfXr1+Pbb7+FnZ0dPvzwQ5WPY2dnB6FQiLRKK2SmpaXByclJ4T5OTk4qlz916hTi4+Px1ltvVVsPkUgES0tLuZs+c3ZWvZyBAeDjA5iZ8dtSU4Fz54BffwW++QZ49Ohl+Z07gVGjXnbpVKSubh6iedSVRghpKOoUhCQlJaF169YAgAMHDmDEiBGYPn06IiMjcerUKZWPY2RkhM6dO8sNGJVIJDh+/DgClWTiCgwMrDLA9OjRowrL//jjj+jcuTP8/PxUrlND0KsX4Ob2chBqZRwHuLvz5UaN4ls7nj8HMjP5RJV79wIrVgCzZvEBilR0NL9MhzIVu3mIfqKuNEJIg8LqwN7enl25coUxxpi/vz/btm0bY4yx+/fvMzMzs1od69dff2UikYht2bKF3b59m02fPp1ZW1uz1NRUxhhjb775Jps3b56s/JkzZ5iBgQFbvnw5i4uLYwsXLmSGhobs5s2bcsfNzc1lpqambMOGDbW+vtzcXAaA5ebm1npfbdm3jzGO4298eMDfpNv27av9MX/8Uf5Yym47dqj/ekj9SV8TlZ+v+rwmCCGktmrzGVqnlpDXX38db731Ft566y3cvXsXgwYNAgDcunULnp6etTpWWFgYli9fjgULFsDf3x/Xrl3DkSNHZINPExMTkVJhEESPHj2wY8cObNq0CX5+fti7dy8OHDiADh06yB33119/BWMMY8eOrcsl6r3QUL5Fw9VVfrubW92n57ZsqVo5VbuDiPZIZ0xRVxohpCGpU56QnJwcfP7550hKSsJ7772HAQMGAAAWLlwIIyMjfPbZZ2qvqDY1hDwhUmIx3z2SksIHB7161X2ZDbGYb7pPTlb8YcZxfJBz4QJw9SowcGC9qk7UKDqa73qpyYkTwKuvaro2hJCmrDafoTpPVqaPGlIQom7S2TGAfCBSMQnarl3A7t3AxInA//0fYGOj/XqSlxgDFi8GFi2queyOHfzMKEII0RSNJys7cuQITp8+Lbu/fv16+Pv7Y9y4ccjOzq7LIYmeqKmbZ9gwftCrQABs2wa0bw/s36+bujZ1ZWX8LKeAANUCEIC60ggh+qVOQcjcuXNlWUVv3ryJjz76CIMGDUJCQgLCw8PVWkGifaGh/NTdEyf4b84nTgAJCfx2oRBYvhw4cwbw9gbS0vjtYWGAkuV+iJrl5vKzm1q14ls1Ll4EjIwAKyvVZkwRQoi+qFN3jLm5OWJjY+Hp6YlFixYhNjYWe/fuxZUrVzBo0CCl2UsbiqbcHVMbRUXAkiV8vhGxGLC1BQ4d4r+Za5s6x8boq9JS4NNPgf/9j59yDQD29sCMGcB77wGnTyvuSpPat09/1xNqCs8fIU2FxrtjjIyMUFhYCAA4duwY+vfvDwCwsbHR+3VXiPoYGwNffcUPVO3Ykc/I6u2t/Xo0ldwYhobA+fN8ANK+PR+MJCYCCxcCDg7Ku9KkXFy0W19VNZXnjxBSVZ1aQoYOHYqSkhL07NkTS5YsQUJCAlxdXfH3339j5syZuHv3ribqqjXUElJ7JSV8F07btvx9xoA//+Rn0CjrIlAH6UDayq9ida4mrKlv6dUdVywGfv8d+O47fiCwdPDvqVNAQQGfil/Z37XycX/8EfjrL2DLFqB8Nr3e0MbzRwjRrlp9htYlEcnjx4/Z4MGDWceOHdn//vc/2fY5c+awWbNm1eWQeqUhJCvTd//7H58oKziYscePX24vK2PsxAk+4dmJE/z9uiorY8zNTXlSNY5jzN29fufYt6/qOdzc6p/4S9lxf/mFsbVrGWvV6uX2r76q37myshjLzKzfMTRBG88fIUT7avMZSlN0FaCWkPpbvx746COguBgwNweWLQPs7IAPP5RfHM/NDVi9um7fdv/+m28RqMkXXwDduwMWFvzN0lK1xGya+pau7LiVNWvGj/WYObNxzmqh3CaENE61+Qw1qOtJxGIxDhw4gLi4OACAj48Phg4dCiGNJiPgB0sGBQFTpwJnz/IfpopIV+dV9IFeVMSPO5Fatgy4fBl4/Jjv+lF1/POSJfL3zc1fDuwEgNGjgZiYl0GKhQVf5ujR6jOQTpvGd3kYGvLdKEIhMGnSyyDl3Dn+celj0rfGu+9WH4AYGPD5V6ZMebnwoDowxud3iYriF7cT1GlEmPrUZjVoQkjjVKeWkPv372PQoEFITk6Gl5cXACA+Ph7u7u44dOgQWrVqpfaKahO1hKiPWAysXQuEh1f/wWttzX+AJybyQcbjx3wAUrHVpE8f4OTJ2tfBz48PAPLz+eDD3ByoOGypZ08+UKovjpNfAHDEiLoPrtTEt//UVH5ab2Eh8MMPQA2LS2sctYQQ0jhpPGPqoEGDwBjD9u3bYVM+Yi4rKwsTJkyAQCDAoUOH6lZzPUFBiHqp+mFTGcfxrSFGRvz9X37hc5F4ePCzJ9zcgC5dak4zn5BQ/UDSR4/4FYalQcrz58A///ADOmvStSs/60Qs5utw8ODLx+bP569dLOaDE7GYP09SUs3H1VRm05Ur+W4ya2vg9m3ddvPcvQu0a6c8OFX1+SOE6BeNByFmZmY4d+4cfH195bZfv34dPXv2RH5+fm0PqVcoCFGvnTv5qZc1GTQIGDDgZZDh4cEn4KqOKmnm6zJuQ1Pf0nX97b+sjB8fc/ky/3fbs0f951BFXh4QGMgHQgD/fCkae0OzYwhpeDSeJ0QkEuF5xU71cvn5+TCSfm0lpJyq37bnzgVmzQKGDuXzjtQUgACaWU0Y4KfLurmpPwOppo6rKgMDPr+IUMj/fX7/XTPnqY5YDIwfzwcgLi5815Ci3CYRERSAENLY1SkIeeONNzB9+nScP38ejDEwxnDu3Dm8++67GDp0qLrrSBo4TX/wVpdmvq6EQn7WjrR+lesLAKtW1b6bQFPHrQ1/f75LBgDef59vldCmzz7ju62MjYEDB/ixKRWfv7Awvtxvv/EBCyGkEavLHODs7Gw2dOhQxnEcMzIyYkZGRozjOBYSEsKys7Prcki9QnlC1G/fPj7vA8dVzQXBcfXPu6EpivJ5uLtrJk+IOo6rqoKCl7lIfvlFO+dkjLGff355vdu3Ky7z7BljdnaMTZnCGL0FCWl4tJYn5P79+7Iput7e3mjdurWaQiPdojEhmhEVBcyeLT/jxd2d/+avz83uusiYqg2nT/OZbl97TTvne/ECaNOGH0gcEQEsXaq8bG6uat1xhBD9o5GBqbVZHXflypUql9VHFIRojq4/eIluPXzIJ7Jbtkz3eUoIIZqhkWRlV69eVakcp8mFQkiDJxRSzgd99Pgxvzje6NGaPU/LlsCKFaqXj48HFi/mW8vs7TVWLUKIjqgchJw4cUKT9SCE6Mj9+/xg1dJSflZSu3bqOzZj/ODXIUPqtnjepEl8cGRn93JALyGk8aAGUUKauFat+Gy0JSXA9OnyWV/r6+uvgY0bgeHD+bEgtfXVV/zP777jgyVCSONCQQghTRzHARs28OvUnDrF5+1Qh99/56fjAnx3iqJcIDXp149PYFdW9vJYhJDGg4IQQgiaN3/Z6vDJJ8DTp/U73s2bfEIyxvjFC5UtYKiKb77hA6Xdu/muGUJI40FBCCEEADBzJtCtG5+8bNasuh8nM5PPepufz6eor+9Yjo4d+bEhAB8g1T2pACFE31AQQggBwM9c+uEHPrV7VBRQl3UoS0r4NWkePeJnwuzZAxga1r9uixfzGVZPngQOH67/8Qgh+kEvgpD169fD09MTxsbGCAgIwIULF6otv2fPHrRr1w7Gxsbw9fXFYQX/leLi4jB06FBYWVnBzMwMXbt2RWJioqYugZBGoWNHfvXfiIi6JzHz8gIsLPgxIba26qmXuzvwxRf8QFdtJVcjhGhevTKmqsOuXbswceJEbNy4EQEBAVi1ahX27NmD+Ph4ODg4VCl/9uxZ9O7dG5GRkXjjjTewY8cOfPPNN7hy5Qo6dOgAAHjw4AG6deuGadOmYezYsbC0tMStW7fQvXt3hcesjJKVEVJ3jPF5Rzw9dV0TQoguaCRjqqYEBASga9euWLduHQBAIpHA3d0ds2bNwrx586qUDwsLQ0FBAQ4ePCjb1r17d/j7+2Pjxo0AgDFjxsDQ0BA///xznepEQQghPLGYz27r5lZ9uTt3gNat+a4cbdWrrAwQibRzPkKI6mrzGarT7piSkhJcvnwZQUFBsm0CgQBBQUGIiYlRuE9MTIxceQAIDg6WlZdIJDh06BDatm2L4OBgODg4ICAgAAcOHFBaj+LiYuTl5cndCGnqHj4EAgOB/v2B4mLl5R48AHr2BAYP5td80bToaD652rJlmj8XIUSzdBqEZGZmQiwWw9HRUW67o6MjUlNTFe6Tmppabfn09HTk5+fj66+/xoABA/D3339j+PDhCA0Nxb///qvwmJGRkbCyspLd3N3d1XB1hDRszZoBiYlAXBw/FkORvDx+JsyzZ0B2NmBkpPl6paYCsbH81N30dM2fjxCiOXoxMFWdJOXpHocNG4YPP/wQ/v7+mDdvHt544w1Zd01lERERyM3Nld2SkpK0WWVC9FKzZsCaNfzvX30F3L4t/7hYzOcCuX0bcHEBDhwATEw0X6/Ro4EuXfgpwIsXa/58hBDN0WkQYmdnB6FQiLS0NLntaWlpcHJyUriPk5NTteXt7OxgYGCA9u3by5Xx9vZWOjtGJBLB0tJS7kYIAUaNAt54g19X5q23gH/+AXbu5LtEIiKAgwf5qbMHDvCBiDYIBMC33/K/f/89cPeuds5LCFE/nQYhRkZG6Ny5M44fPy7bJpFIcPz4cQQGBircJzAwUK48ABw9elRW3sjICF27dkV8fLxcmbt378LDw0PNV0BI48Zx/LotxsZATAyfRn3cOD4JmXRMxo8/Al27ardeffvyC+KVlfFTigkhDZOWxrIrFx4ejkmTJqFLly7o1q0bVq1ahYKCAkyZMgUAMHHiRLi6uiIyMhIAMHv2bPTp0wcrVqzA4MGD8euvv+LSpUvYtGmT7Jhz585FWFgYevfujb59++LIkSP4448/EB0drYtLJKRBu3gRKCpS/rixsfbqUtE33wBHjgD79vEBkpLvLYQQfcb0wNq1a1nz5s2ZkZER69atGzt37pzssT59+rBJkybJld+9ezdr27YtMzIyYj4+PuzQoUNVjvnjjz+y1q1bM2NjY+bn58cOHDigcn1yc3MZAJabm1vnayKkMSgrY8zNjTE++0fVG8cx5u7Ol9OFqVP5esycqZvzE0Kqqs1nqM7zhOgjyhNCCC86mu/6qMmJE8Crr2q6NlUlJwMXLgAhIXzXESFE92rzGarz7hhCiP5KSVFvOXVzdQWGD9fNuQkh9dfopugSQtTH2Vm95TQpNxc4f17XtSCE1AYFIYQQpXr14lO2K+vq4Dh+cblevbRbr8quXwdatQKGDePzhxBCGgYKQgghSgmFwOrV/O+VAxHp/VWr+HK65O0NWFsDaWnAihW6rQshRHUUhBBCqhUaCuzdy4+/qMjNjd8eGqqbelVkZASUz+LHsmV8andCiP6j2TEK0OwYQqoSi4FTp/hBqM7OfBeMrltAKmIM6N6dny3z7rvAhg26rhEhTVNtPkMpCFGAghBCGqaTJ4E+ffjg6NYtwMtL1zUipOmpzWcodccQQhqN3r35VX3FYn5tG0KIfqMghBDSqHz9NWBoCFhZ8WvLEEL0FyUrI4Q0Kt7ewOPH/LgVsZjP+qqv41gIaeooCCGENDrOzkBUFDB7NvDkycvtbm78lGN9mNFDCKHuGEJIIxQVBYwcKR+AAPxaMyNH8o/rI2nLzc6d/E+xWNc1IkSzKAghhDQqYjHfAqJo3p9025w5+vcBHxUFeHryCwaOG8f/9PTU34CJEHWgIIQQ0qicOlW1BaQixoCkJL6cvmioLTeE1BcFIYSQRkXfV/6trKG23BCiDhSEEEIaFVVX9DU21mw9VNUQW24IURcKQgghjUpNK/8CgJkZn9RMqrBQ8/VS5tYt1crpS8sNIepEQQghpFGpaeVfjgO2bHmZLyQ2lg9avvoKKCjQXj3PngVGjwY++EC18qq28BDSkFAQQghpdGpa+XfkyJfbfvwRyM4GPv8caNUK+O47oLRU83WMigL27AEkEn4V4Jr8/DPw/Lnm60WINtECdgrQAnaENA6qrPwrkQC7dvFByMOH/LZWrYD//pdvqRCo4ava3bt8cBMayq9vA/Dn+vprYMYM4MGDl4FRxf/IHCd/v0ULYNs24D//qX+dCNEUWkW3nigIIaTpKSkBfvgBWLwYSE/ntwUFAX//XbVbR5XgRiwGDh8G1q3jjwEAw4crn26rKMOruzuwahVgawtMnAgkJvJ1+fRT4MsvVWtBIUTbKAipJwpCCGm68vP5D/5vvwUWLQLCw+UfrykdfFYW38WzYQPw6BH/OMcBgwcDs2YB/fsrP3d1wU1uLn/erVv5+35+fJDj4qKmCydETSgIqScKQgghmZmAufnLqby//cZ3n5w/XzWnh7SlZO9evhvn6lX+frNmwFtvAe+9x3elqMP+/cD06fzxzpzhVwwmRJ/U5jOUFrAjhBAF7Oxe/s4Y8NlnyqfTMsYHInPmAB9/zG+bORMYMwYwNVVvvYYPBwID+WnF0gCkuBhISwOaN1fvuQjRNJodQwghNeA44KOPqi8jTSrm4wNcvgxMnar+AETKyQlo2fLl/UWLAF9fvquG2rZJQ6IXQcj69evh6ekJY2NjBAQE4MKFC9WW37NnD9q1awdjY2P4+vri8OHDco9PnjwZHMfJ3QYMGKDJSyCENHKqZlhNT68+UZq6lZXx3TJ5ecDkycCIEUBGhvbOT0h96DwI2bVrF8LDw7Fw4UJcuXIFfn5+CA4ORrp0eHolZ8+exdixYzFt2jRcvXoVISEhCAkJQWxsrFy5AQMGICUlRXbbuXOnNi6HENJIqZosTNtJxQwMgBMngMhIvntm/36+VeTgQflyYjEQHQ3s3Mn/pLVoiD7Q+cDUgIAAdO3aFevWrQMASCQSuLu7Y9asWZg3b16V8mFhYSgoKMDBCu+w7t27w9/fHxs3bgTAt4Tk5OTgwIEDdaoTDUwlhFQmFgOenvzKtor+a3IcP0smIaHqdF1tuXYNmDDh5diVt98GVq7kpwhXN6OHEHWqzWeoTltCSkpKcPnyZQQFBcm2CQQCBAUFISYmRuE+MTExcuUBIDg4uEr56OhoODg4wMvLC++99x6ysrKU1qO4uBh5eXlyN0IIqaimdPAAP7VXVwEIAPj7A5cu8dOKOY5v9fj5Zz4RWuVF8pKT+e3K8pYQog06DUIyMzMhFovh6Ogot93R0RGpqakK90lNTa2x/IABA7Bt2zYcP34c33zzDf79918MHDgQYiXtj5GRkbCyspLd3N3d63llhJDGqKZ08PrQqmBsDKxYAfzzD7BpE7B0qeKWG+m2OXOoa4boTqOcojtmzBjZ776+vujYsSNatWqF6Oho9OvXr0r5iIgIhFfISJSXl0eBCCFEodBQYNiwmjOm6tqrr/JjPyq3gFQkndFz6hRfnhBt02kQYmdnB6FQiLS0NLntaWlpcHJyUriPk5NTrcoDQMuWLWFnZ4f79+8rDEJEIhFEIlEdroAQ0hQJhQ3jQzslRb3lCFE3nXbHGBkZoXPnzjh+/Lhsm0QiwfHjxxEYGKhwn8DAQLnyAHD06FGl5QHgyZMnyMrKgjOthU0IaUL0dUYPIVI6n6IbHh6OH374AVu3bkVcXBzee+89FBQUYMqUKQCAiRMnIiIiQlZ+9uzZOHLkCFasWIE7d+5g0aJFuHTpEmbOnAkAyM/Px9y5c3Hu3Dk8evQIx48fx7Bhw9C6dWsEBwfr5BoJIUQXevXix6soy1vCcfwieb16abdehEjpfExIWFgYMjIysGDBAqSmpsLf3x9HjhyRDT5NTEyEoMJa2j169MCOHTvw+eefY/78+WjTpg0OHDiADh06AACEQiFu3LiBrVu3IicnBy4uLujfvz+WLFlCXS6EkCZFOqNn5Eg+4FA0QFXXM3pI06bzPCH6iPKEEEIaE0Ur/3Ic8P33fC4RQtSpweQJIYQQonmhocCjR3xm1e3b+fVtGAPy83VdM9LU6bw7hhBCiOZVnNHTvj2/6i4NkyO6RkEIIYQ0Mf7+uq4BITzqjiGEkCYsLQ04e1bXtSBNFbWEEEJIE3X+PPD664C5OXD3Lv+TEG2ilhBCCGmi/PwAe3s+Y2pkpK5rQ5oiCkIIIaSJki52B/A/ExJ0Wx/S9FAQQgghTdiwYcBrrwHFxcDcubquDWlqKAghhJAmjOP4rKkCAbBvH7/yLiHaQkEIIYQ0cb6+wDvv8L/PmQOIxTqtDmlCKAghhBCCxYv5Qaqvvsp3zeiaWMy3yuzcyf+kwKhxoim6hBBCYGcHPHyoH9N0Fa114+bGL8YXGqq7ehH1o5YQQgghAPQnABk5Uj4AAYDkZH57VJRu6kU0g4IQQgghcm7cAPr3B+7c0e55xWK+BUTR2u7SbTRmpXGhIIQQQoicL74Ajh4FwsO1e95Tp6q2gFTEGJCUxJfTRzSOpfYoCCGEECJn2TLA0BD480/g8GHtnTclRb3ltCkqCvD0BPr2BcaN4396elL3UU0oCCGEECKnbVvggw/438PDgdJS7ZzX0VG1cs7Omq1HbdE4lrqjIIQQQkgVX3zBT9mNjwfWr9f8+V68ADZsqLmcmRkQGKj5+qiKxrHUDwUhhBBCqrCyApYu5X9ftAjIyNDcuTIygH79gL17AaGQ38ZxissWFACjRgGFhZqrT2009HEsukZBCCGEEIWmTAH8/YHcXGDdOs2cIzER6N4diIkBmjUDjh/n08e7usqXc3cHPvmEX3Tvjz/49W4yMzVTp9pISlKtnD6OY9EHlKyMEEKIQkIhsHYtcO0a8O67mjmHoyPg4sK3GBw+DLRrx28fNoxvPUhJ4ceA9OrF12foUGDIEOD8eeD114HLl/l1b3Th4UMgMlK1svo2jkVfcIwp6slq2vLy8mBlZYXc3FxYWlrqujqEENLoMPayyyUrix8z4eCg2r5xcXwgsmoV8MYbGqtitbZvB957D3j+nL+O6j5J3d2BhISXXU2NXW0+Q6k7hhBCiEqKi/mBqvXBGPDf/wLz5r3cZmuregACAN7ewO3b8gHIixf1q5eqnj8HJk4EJkzgf//Pf/gBtRynfBxLhw66a63Rd/RnIYQQUqPbtwEfH2DAAKCoqG7HKCkBpk7lZ958+y0/DqSujIxe/v7gAT+tePv2uh9PFRcuAK+8Avz8Mx9UfPklcOIEvwLx3r1Vx7HY2vI///xT83VrqCgIIYQQUiMPDz74ePQIWLmy9vvn5AADBwJbtvAf4N99p76ptps28TNUJkzggxtNDTJYtowPeJo3B06eBBYsAAzKR1aGhvJ/mxMngB07+J9pafx1jh8PhIVppk4NHtMD69atYx4eHkwkErFu3bqx8+fPV1t+9+7dzMvLi4lEItahQwd26NAhpWXfeecdBoD93//9n8r1yc3NZQBYbm6uyvsQQkhjt307YwBjZmaMJServt+jR4y1b/9y32r+ZdeJWMzYRx/xxwcYmzWLsbIy9Z6DMcYyMxl7/33GsrNrt59Eov666LPafIbqvCVk165dCA8Px8KFC3HlyhX4+fkhODgY6enpCsufPXsWY8eOxbRp03D16lWEhIQgJCQEsbGxVcru378f586dg4uLi6YvgxBCGr2xY/nWi4ICICJCtX0uXQICAvjuHBcX4PRpYNAg9dZLIACWL3/ZQrN2LTBmTN27jaR++w2YOfNly4qtLZ+4zdq6dseRjhURi/njHThQv3o1KloIiqrVrVs3NmPGDNl9sVjMXFxcWGRkpMLyo0ePZoMHD5bbFhAQwN555x25bU+ePGGurq4sNjaWeXh4VNsSUlRUxHJzc2W3pKQkagkhhBAFLlx42eJw7lzN5Xfv5st27MhYUpLm6/frr4wZGfHn7NWr9q0WjDFWWMi3eEivc98+9dRt0yb+eEZGjB07pp5j6qMG0xJSUlKCy5cvIygoSLZNIBAgKCgIMUpGLMXExMiVB4Dg4GC58hKJBG+++Sbmzp0LHx+fGusRGRkJKysr2c3d3b2OV0QIIY1b167ApEn877NnAxJJ9eVHjQJ27+Zzfri5ab5+YWHAkSOApSXfEmJoWLv9Y2OBbt34sRwA8NFHwODB6qnblCn82JGSEj4PyoUL6jluQ6bTICQzMxNisRiOlVYtcnR0RGpqqsJ9UlNTayz/zTffwMDAAB9IV2CqQUREBHJzc2W3JFVT4BFCSBMUGQmYmwM2NkB2tvzy9SUlwMKF8qnMR43igwJt6duX7/Y5eJBfa0YVjPFdLV268IGIoyMfzCxfDohE6qmXgQE/aLVfP75La+BAvpuqKWt0GVMvX76M1atX48qVK+CUTdquRCQSQaSuVxkhhDRyzs7AzZvAlSt8WveKAYexMd8C8ccf/Dd9Ax19yvj6yt9fvpwPMHr1UpyJ9d13+Vk2wMtZPLXJXaIqkYgfExIU9DLr65kzgKen+s/VEOg0CLGzs4NQKERaWprc9rS0NDg5OSncx8nJqdryp06dQnp6Opo3by57XCwW46OPPsKqVavw6NEjtdVfLBajVFtrXBO1MzQ0hLCppDAkRM2uXOGXqa88HVY6GLRfP90FIJX9/jswdy5fH0tL4Nmzl4+5uQGrVwOjRwPbtvGtPB98oNnkYubmwKFDQJ8+wK1bfNBz40btu44aA52nbQ8ICEC3bt2wdu1aAPx4jubNm2PmzJmYVzGlXrmwsDAUFhbijz/+kG3r0aMHOnbsiI0bNyIrKwsplVYKCg4OxptvvokpU6bAy8urxjrVlHKWMYbU1FTk5OTU8mqJvrG2toaTk5PKrWaEEH6Wh6dn9avH6lOq8qIi4NVX+ZaHyqRv/b17gZ49+W4YbXn6lG8JiYzk18RpLGqTtl3ncWp4eDgmTZqELl26oFu3bli1ahUKCgowZcoUAMDEiRPh6uqKyPJVgmbPno0+ffpgxYoVGDx4MH799VdcunQJm8rb0WxtbWErTVNXztDQEE5OTioFIKqQBiAODg4wNTWlD7AGiDGGwsJC2VRwZ1pdihCV1bR8PfBy+fpXX9VKlaplaAgkJyt+TLqGzZw5fNCkTS4uwPXr+tNipAs6v/SwsDBkZGRgwYIFSE1Nhb+/P44cOSIbfJqYmAhBhXaxHj16YMeOHfj8888xf/58tGnTBgcOHECHDh20Ul+xWCwLQCoHO6RhMTExAQCkp6fDwcGBumYIUZGqy9Lry/L1NQVNjOkuaKoYgCQk8K0ia9eqbzCsvtN5EAIAM2fOxMyZMxU+Fh0dXWXbqFGjMGrUKJWPr85xINIxIKampmo7JtEd6fNYWlpKQQghKlK14VBfGhgbQtBUVgYEBwP37vEp7nfu1I+uLE3TecbUhoq6YBoHeh4Jqb1evfgBncrePhzHjwnp1Uu79VKmIQRNBgbAunV819GePcB772luDRx9QkEIIYSQWhEK+RklQNVARHp/1Sr9+SbfUIKm/v35PCICAfDDD6qnxm/IKAghhBBSa6Ghipevd3Pjt4eG6qZeijSkoGnkSOD77/nfv/mGv4nF8gnhxGJd1lC9KAjRoYb8wvL09MSqVavUcqzo6GhwHEdTnglpYBQtX5+QoF8BiFRDCpreegv49lv+93nz+KRpffsC48bxPz09gagonVZRbfRiYGpTFBXFr7tQccS2NGmOpt4Mr776Kvz9/dUSPFy8eBFmquZDJoQ0WkKhfkzDVUVoKL9mi6KMqfpm7lzg3Dn+s6JicjWAn248cqT+BU91QUGIDkRFKc40qOsXFmMMYrEYBipMWre3t9dCjQghRL0aStAkFitf4K5ibpNhw/QziFIVdceoUUGB8ps0lbFYzLeAKBr1LN02e7Z814yi49XW5MmT8e+//2L16tXgOA4cx2HLli3gOA5//vknOnfuDJFIhNOnT+PBgwcYNmwYHB0dYW5ujq5du+LYsWNyx6vcHcNxHP73v/9h+PDhMDU1RZs2bfD777/XvqLl9u3bBx8fH4hEInh6emLFihVyj3/33Xdo06YNjI2N4ejoiJEjR8oe27t3L3x9fWFiYgJbW1sEBQWhoC5/NEII0ZHa5DZpyCgIUSNzc+W3ESP4Mqq8sJ48kX9heXpWPV5trV69GoGBgXj77beRkpKClJQUuLu7AwDmzZuHr7/+GnFxcejYsSPy8/MxaNAgHD9+HFevXsWAAQMwZMgQJCYmVnuOL7/8EqNHj8aNGzcwaNAgjB8/Hs8qtyOq4PLlyxg9ejTGjBmDmzdvYtGiRfjiiy+wZcsWAMClS5fwwQcfYPHixYiPj8eRI0fQu3dvAEBKSgrGjh2LqVOnIi4uDtHR0QgNDYWOVycghJBaaQi5TdSBumO0TFcvLCsrKxgZGcHU1FS22N+dO3cAAIsXL8brr78uK2tjYwM/Pz/Z/SVLlmD//v34/ffflSaVA/jWlrFjxwIAli5dijVr1uDChQsYMGBAreq6cuVK9OvXD1988QUAoG3btrh9+zaWLVuGyZMnIzExEWZmZnjjjTdgYWEBDw8PvPLKKwD4IKSsrAyhoaHw8PAAAPhWXk6TEEL0XEPIbaIO1BKiRvn5ym/79vFl6vLCevSo6vHUqUuXLnL38/Pz8fHHH8Pb2xvW1tYwNzdHXFxcjS0hHTt2lP1uZmYGS0tL2dostREXF4eePXvKbevZsyfu3bsHsViM119/HR4eHmjZsiXefPNNbN++HYWFhQAAPz8/9OvXD76+vhg1ahR++OEHZGdn17oOhBCiSzXlNgH0I7dJfVEQokZmZspvxsZ8mbokzVF0PPXWW/6AH3/8Mfbv34+lS5fi1KlTuHbtGnx9fVFSUlLtcQwrrUPNcRwkEol6KwvAwsICV65cwc6dO+Hs7IwFCxbAz88POTk5EAqFOHr0KP7880+0b98ea9euhZeXFxK0vTIVIYTUQ3W5TaQ++qhhD0oFKAjROl0mzTEyMoJYhWQkZ86cweTJkzF8+HD4+vrCyclJrevv1MTb2xtnzpypUqe2bdvK1ncxMDBAUFAQvv32W9y4cQOPHj3CP//8A4APfnr27Ikvv/wSV69ehZGREfbv36+1+hNCiDooy20iXdzu++/V3zKubTQmRAekLyxFeUJWrdLc9FxPT0+cP38ejx49grm5udJWijZt2iAqKgpDhgwBx3H44osvNNKiocxHH32Erl27YsmSJQgLC0NMTAzWrVuH7777DgBw8OBBPHz4EL1790azZs1w+PBhSCQSeHl54fz58zh+/Dj69+8PBwcHnD9/HhkZGfD29tZa/QkhRF0U5TZp2xbo0gWIiwN+/51PYtZQURCiI7pImvPxxx9j0qRJaN++PV68eIGffvpJYbmVK1di6tSp6NGjB+zs7PDpp58iLy9PcxWrpFOnTti9ezcWLFiAJUuWwNnZGYsXL8bkyZMBANbW1oiKisKiRYtQVFSENm3aYOfOnfDx8UFcXBxOnjyJVatWIS8vDx4eHlixYgUGDhyotfoTQog6KcptsmsXkJ7+cuZlQ8UxmrtYRV5eHqysrJCbmwtLS0u5x4qKipCQkIAWLVrAWDrQgzRY9HwSQoh6VfcZWhmNCSGEEEIasORkYMIEoCEuv0VBCNG4d999F+bm5gpv7777rq6rRwghDRZjfJfM9u3AlCmKs3HrMxoTQjRu8eLF+PjjjxU+VlNTHSGEEOU4Dli/HujRAzhwAFi5kp+621BQEEI0zsHBAQ4ODrquBiGENEqdO/MzK99/H/j0U6B7d6BSvke9Rd0xhBBCSAP37rvAmDH84qdhYUBGhq5rpBoKQgghhJAGjuOATZsALy9+oOr48fKrsesrCkIIIYSQRsDCgk+EaWICPH0KZGbqukY1ozEhhBBCSCPRoQPw119Ap07qX2dMEygIIYQQQhqRyivrisX6u9CdXnTHrF+/Hp6enjA2NkZAQAAuXLhQbfk9e/agXbt2MDY2hq+vLw4fPiz3+KJFi9CuXTuYmZmhWbNmCAoKwvnz5zV5CXUiZgzR2dnYmZaG6OxsiPV8grenpydWrVqlUlmO43DgwAGN1ocQQohyEgnw9ddAv35Aaamua6OYzoOQXbt2ITw8HAsXLsSVK1fg5+eH4OBgpKenKyx/9uxZjB07FtOmTcPVq1cREhKCkJAQxMbGysq0bdsW69atw82bN3H69Gl4enqif//+yNCj4cJRGRnwPHcOfa9fx7i4OPS9fh2e584hSo/qSAghpOF6+hSIjAT+/Rf47DNd10YxnQchK1euxNtvv40pU6agffv22LhxI0xNTbF582aF5VevXo0BAwZg7ty58Pb2xpIlS9CpUyesW7dOVmbcuHEICgpCy5Yt4ePjg5UrVyIvLw83btzQ1mVVKyojAyNv3cKT4mK57cnFxRh56xYFIoQQQurNzQ2QrlO6bBm/4q6+0WkQUlJSgsuXLyMoKEi2TSAQICgoCDExMQr3iYmJkSsPAMHBwUrLl5SUYNOmTbCysoKfn5/CMsXFxcjLy5O71QZjDAVisUq3vLIyfHDvHhR1vEi3zb5/H3llZTUeqzZrD27atAkuLi6QSCRy24cNG4apU6fiwYMHGDZsGBwdHWFubo6uXbvi2LFjtfo7VOfmzZt47bXXYGJiAltbW0yfPh35+fmyx6Ojo9GtWzeYmZnB2toaPXv2xOPHjwEA169fR9++fWFhYQFLS0t07twZly5dUlvdCCGksQoNBebM4X+fNAlISNBpdarQ6cDUzMxMiMViODo6ym13dHTEnTt3FO6TmpqqsHxqaqrctoMHD2LMmDEoLCyEs7Mzjh49Cjs7O4XHjIyMxJdfflnn6yiUSGB+6lSd96+IAXhSXAyr06drLJvfqxfMVBxtNGrUKMyaNQsnTpxAv379AADPnj3DkSNHcPjwYeTn52PQoEH46quvIBKJsG3bNgwZMgTx8fFo3rx5fS4JBQUFCA4ORmBgIC5evIj09HS89dZbmDlzJrZs2YKysjKEhITg7bffxs6dO1FSUoILFy6A4zgAwPjx4/HKK69gw4YNEAqFuHbtGgwNDetVJ0IIaSq++QY4d46/jRoFnD4N6Mui4TrvjtGUvn374tq1azh79iwGDBiA0aNHKx1nEhERgdzcXNktKSlJy7XVvGbNmmHgwIHYsWOHbNvevXthZ2eHvn37ws/PD++88w46dOiANm3aYMmSJWjVqhV+V0P73Y4dO1BUVIRt27ahQ4cOeO2117Bu3Tr8/PPPSEtLQ15eHnJzc/HGG2+gVatW8Pb2xqRJk2TBT2JiIoKCgtCuXTu0adMGo0aNUtqqRQghRJ6REbB7N2BrC1y+DISH67pGL+m0JcTOzg5CoRBpaWly29PS0uDk5KRwHycnJ5XKm5mZoXXr1mjdujW6d++ONm3a4Mcff0RERESVY4pEIohEojpfh6lAgPzKc6KUOJmTg0E3b9ZY7rCvL3pbW9d43toYP3483n77bXz33XcQiUTYvn07xowZA4FAgPz8fCxatAiHDh1CSkoKysrK8OLFCyQmJtbqHIrExcXBz88PZhUmrffs2RMSiQTx8fHo3bs3Jk+ejODgYLz++usICgrC6NGj4ezsDAAIDw/HW2+9hZ9//hlBQUEYNWoUWrVqVe96EUJIU+HuDvzyCxASAvj46Lo2L+m0JcTIyAidO3fG8ePHZdskEgmOHz+OwMBAhfsEBgbKlQeAo0ePKi1f8bjFlQaCqgvHcTATClW69bexgZtIBE7ZsQC4i0Tob2NT47Gk3RWqGjJkCBhjOHToEJKSknDq1CmMHz8eAPDxxx9j//79WLp0KU6dOoVr167B19cXJSUl9fvjqOinn35CTEwMevTogV27dqFt27Y4d+4cAH7K9a1btzB48GD8888/aN++Pfbv36+VehFCSGMxYAA/JmTGDP6+WAxERwM7d/I/dZHmXefdMeHh4fjhhx+wdetWxMXF4b333kNBQQGmTJkCAJg4caJc68Xs2bNx5MgRrFixAnfu3MGiRYtw6dIlzJw5EwA//mD+/Pk4d+4cHj9+jMuXL2Pq1KlITk7GqFGjdHKNFQk5DqtbtwaAKoGI9P6q1q0hrGWAoQpjY2OEhoZi+/bt2LlzJ7y8vNCpUycAwJkzZzB58mQMHz4cvr6+cHJywqNHj9RyXm9vb1y/fh0FBQWybWfOnIFAIICXl5ds2yuvvIKIiAicPXsWHTp0kOs6atu2LT788EP8/fffCA0NxU/SId+EEEJUVt7AjKgooHlzoG9fYNw4/qenJ79dm3QehISFhWH58uVYsGAB/P39ce3aNRw5ckQ2+DQxMREpKSmy8j169MCOHTuwadMm+Pn5Ye/evThw4AA6dOgAABAKhbhz5w5GjBiBtm3bYsiQIcjKysKpU6fgoydtUKH29tjr4wPXSl1AbiIR9vr4INTeXmPnHj9+PA4dOoTNmzfLWkEAoE2bNoiKisK1a9dw/fp1jBs3rspMmvqc09jYGJMmTUJsbCxOnDiBWbNm4c0334SjoyMSEhIQERGBmJgYPH78GH///Tfu3bsHb29vvHjxAjNnzkR0dDQeP36MM2fO4OLFi/D29lZL3QghpKmJigJGjuTziFSUnMxv12ogwkgVubm5DADLzc2t8tiLFy/Y7du32YsXL+p9njKJhJ149oztSE1lJ549Y2USSb2PWROxWMycnZ0ZAPbgwQPZ9oSEBNa3b19mYmLC3N3d2bp161ifPn3Y7NmzZWU8PDzY//3f/6l0HgBs//79svs3btxgffv2ZcbGxszGxoa9/fbb7Pnz54wxxlJTU1lISAhzdnZmRkZGzMPDgy1YsICJxWJWXFzMxowZw9zd3ZmRkRFzcXFhM2fOVMvfnzH1Pp+EEKLvysoYc3NjDFB84zjG3N35cnVV3WdoZRxjep4rXAfy8vJgZWWF3NxcWFpayj1WVFSEhIQEtGjRAsb6MseJ1Bk9n4SQpiQ6mu96qcmJE8Crr9btHNV9hlam8+4YQgghhGhHhdENailXXxSEkFrbvn07zM3NFd70ZdwNIYSQqqQDU9VVrr50mieENExDhw5FQECAwscokykhhOivXr34NWWSk/lRIJVxHP+4iqmv6o2CEFJrFhYWsLCw0HU1CCGE1JJQCKxezc+C4Tj5QESaGWLVKr6cNlB3TB3ReN7GgZ5HQkhTExoK7N0LuLrKb3dz47eHhmqvLtQSUkvS7obCwkKYmJjouDakvgoLCwFQNxIhpGkJDQWGDQNOneIHoTo7810w2moBkaIgpJaEQiGsra1li+GZmprWOn060T3GGAoLC5Geng5ra2sItf3OI4QQHRMK6z4NV10oCKkD6WJ5ylblJQ2HtbW10sUSCSGEaBYFIXXAcRycnZ3h4OCA0tJSXVeH1JGhoSG1gBBCiA5REFIPQqGQPsQIIYSQOqLZMYQQQgjRCQpCCCGEEKITFIQQQgghRCdoTIgC0gRWeXl5Oq4JIYQQ0rBIPztVSQZJQYgCz58/BwC4u7vruCaEEEJIw/T8+XNYWVlVW4ZjlLe6ColEgqdPn8LCwqLRJSLLy8uDu7s7kpKSYGlpqevqqFVjvjaArq8ha8zXBjTu62vM1wZo5voYY3j+/DlcXFwgEFQ/6oNaQhQQCARwc3PTdTU0ytLSslG+oYDGfW0AXV9D1pivDWjc19eYrw1Q//XV1AIiRQNTCSGEEKITFIQQQgghRCcoCGliRCIRFi5cCJFIpOuqqF1jvjaArq8ha8zXBjTu62vM1wbo/vpoYCohhBBCdIJaQgghhBCiExSEEEIIIUQnKAghhBBCiE5QEEIIIYQQnaAgpBGJjIxE165dYWFhAQcHB4SEhCA+Pr7afbZs2QKO4+RuxsbGWqqx6hYtWlSlnu3atat2nz179qBdu3YwNjaGr68vDh8+rKXa1p6np2eV6+M4DjNmzFBYXt+ft5MnT2LIkCFwcXEBx3E4cOCA3OOMMSxYsADOzs4wMTFBUFAQ7t27V+Nx169fD09PTxgbGyMgIAAXLlzQ0BUoV921lZaW4tNPP4Wvry/MzMzg4uKCiRMn4unTp9Uesy6vb02p6bmbPHlylboOGDCgxuPqw3MH1Hx9it6HHMdh2bJlSo+pL8+fKp8BRUVFmDFjBmxtbWFubo4RI0YgLS2t2uPW9f2qCgpCGpF///0XM2bMwLlz53D06FGUlpaif//+KCgoqHY/S0tLpKSkyG6PHz/WUo1rx8fHR66ep0+fVlr27NmzGDt2LKZNm4arV68iJCQEISEhiI2N1WKNVXfx4kW5azt69CgAYNSoUUr30efnraCgAH5+fli/fr3Cx7/99lusWbMGGzduxPnz52FmZobg4GAUFRUpPeauXbsQHh6OhQsX4sqVK/Dz80NwcDDS09M1dRkKVXdthYWFuHLlCr744gtcuXIFUVFRiI+Px9ChQ2s8bm1e35pU03MHAAMGDJCr686dO6s9pr48d0DN11fxulJSUrB582ZwHIcRI0ZUe1x9eP5U+Qz48MMP8ccff2DPnj34999/8fTpU4SGhlZ73Lq8X1XGSKOVnp7OALB///1XaZmffvqJWVlZaa9SdbRw4ULm5+encvnRo0ezwYMHy20LCAhg77zzjpprphmzZ89mrVq1YhKJROHjDeV5Y4wxAGz//v2y+xKJhDk5ObFly5bJtuXk5DCRSMR27typ9DjdunVjM2bMkN0Xi8XMxcWFRUZGaqTeqqh8bYpcuHCBAWCPHz9WWqa2r29tUXR9kyZNYsOGDavVcfTxuWNMtedv2LBh7LXXXqu2jL4+f5U/A3JycpihoSHbs2ePrExcXBwDwGJiYhQeo67vV1VRS0gjlpubCwCwsbGptlx+fj48PDzg7u6OYcOG4datW9qoXq3du3cPLi4uaNmyJcaPH4/ExESlZWNiYhAUFCS3LTg4GDExMZquZr2VlJTgl19+wdSpU6tdQLGhPG+VJSQkIDU1Ve75sbKyQkBAgNLnp6SkBJcvX5bbRyAQICgoSO+f09zcXHAcB2tr62rL1eb1rWvR0dFwcHCAl5cX3nvvPWRlZSkt25Cfu7S0NBw6dAjTpk2rsaw+Pn+VPwMuX76M0tJSueeiXbt2aN68udLnoi7v19qgIKSRkkgkmDNnDnr27IkOHTooLefl5YXNmzfjt99+wy+//AKJRIIePXrgyZMnWqxtzQICArBlyxYcOXIEGzZsQEJCAnr16oXnz58rLJ+amgpHR0e5bY6OjkhNTdVGdevlwIEDyMnJweTJk5WWaSjPmyLS56A2z09mZibEYnGDe06Liorw6aefYuzYsdUuDlbb17cuDRgwANu2bcPx48fxzTff4N9//8XAgQMhFosVlm+ozx0AbN26FRYWFjV2V+jj86foMyA1NRVGRkZVAuLqnou6vF9rg1bRbaRmzJiB2NjYGvslAwMDERgYKLvfo0cPeHt74/vvv8eSJUs0XU2VDRw4UPZ7x44dERAQAA8PD+zevVulbykNyY8//oiBAwfCxcVFaZmG8rw1ZaWlpRg9ejQYY9iwYUO1ZRvS63vMmDGy3319fdGxY0e0atUK0dHR6Nevnw5rpn6bN2/G+PHjaxz0rY/Pn6qfAbpGLSGN0MyZM3Hw4EGcOHECbm5utdrX0NAQr7zyCu7fv6+h2qmHtbU12rZtq7SeTk5OVUZ8p6WlwcnJSRvVq7PHjx/j2LFjeOutt2q1X0N53gDInoPaPD92dnYQCoUN5jmVBiCPHz/G0aNHa71Eek2vb33SsmVL2NnZKa1rQ3vupE6dOoX4+PhavxcB3T9/yj4DnJycUFJSgpycHLny1T0XdXm/1gYFIY0IYwwzZ87E/v378c8//6BFixa1PoZYLMbNmzfh7OysgRqqT35+Ph48eKC0noGBgTh+/LjctqNHj8q1Huijn376CQ4ODhg8eHCt9msozxsAtGjRAk5OTnLPT15eHs6fP6/0+TEyMkLnzp3l9pFIJDh+/LjePafSAOTevXs4duwYbG1ta32Mml7f+uTJkyfIyspSWteG9NxV9OOPP6Jz587w8/Or9b66ev5q+gzo3LkzDA0N5Z6L+Ph4JCYmKn0u6vJ+rW2lSSPx3nvvMSsrKxYdHc1SUlJkt8LCQlmZN998k82bN092/8svv2R//fUXe/DgAbt8+TIbM2YMMzY2Zrdu3dLFJSj10UcfsejoaJaQkMDOnDnDgoKCmJ2dHUtPT2eMVb2uM2fOMAMDA7Z8+XIWFxfHFi5cyAwNDdnNmzd1dQk1EovFrHnz5uzTTz+t8lhDe96eP3/Orl69yq5evcoAsJUrV7KrV6/KZoh8/fXXzNramv3222/sxo0bbNiwYaxFixbsxYsXsmO89tprbO3atbL7v/76KxOJRGzLli3s9u3bbPr06cza2pqlpqbqzbWVlJSwoUOHMjc3N3bt2jW592FxcbHSa6vp9a0v1/f8+XP28ccfs5iYGJaQkMCOHTvGOnXqxNq0acOKioqUXp++PHc1XZ9Ubm4uMzU1ZRs2bFB4DH19/lT5DHj33XdZ8+bN2T///MMuXbrEAgMDWWBgoNxxvLy8WFRUlOy+Ku/XuqIgpBEBoPD2008/ycr06dOHTZo0SXZ/zpw5rHnz5szIyIg5OjqyQYMGsStXrmi/8jUICwtjzs7OzMjIiLm6urKwsDB2//592eOVr4sxxnbv3s3atm3LjIyMmI+PDzt06JCWa107f/31FwPA4uPjqzzW0J63EydOKHwtSq9BIpGwL774gjk6OjKRSMT69etX5bo9PDzYwoUL5batXbtWdt3dunVj586d09IVvVTdtSUkJCh9H544cUJ2jMrXVtPrW5uqu77CwkLWv39/Zm9vzwwNDZmHhwd7++23qwQT+vrcMVbza5Mxxr7//ntmYmLCcnJyFB5DX58/VT4DXrx4wd5//33WrFkzZmpqyoYPH85SUlKqHKfiPqq8X+uKKz8hIYQQQohW0ZgQQgghhOgEBSGEEEII0QkKQgghhBCiExSEEEIIIUQnKAghhBBCiE5QEEIIIYQQnaAghBBCCCE6QUEIIYQQQnSCghBCSJMQHR0NjuOqLN5FCNEdCkIIIYQQohMUhBBCCCFEJygIIYRohUQiQWRkJFq0aAETExP4+flh7969AF52lRw6dAgdO3aEsbExunfvjtjYWLlj7Nu3Dz4+PhCJRPD09MSKFSvkHi8uLsann34Kd3d3iEQitG7dGj/++KNcmcuXL6NLly4wNTVFjx49EB8fr9kLJ4QoRUEIIUQrIiMjsW3bNmzcuBG3bt3Chx9+iAkTJuDff/+VlZk7dy5WrFiBixcvwt7eHkOGDEFpaSkAPngYPXo0xowZg5s3b2LRokX44osvsGXLFtn+EydOxM6dO7FmzRrExcXh+++/h7m5uVw9PvvsM6xYsQKXLl2CgYEBpk6dqpXrJ4QooJa1eAkhpBpFRUXM1NSUnT17Vm77tGnT2NixY2XLq//666+yx7KyspiJiQnbtWsXY4yxcePGsddff11u/7lz57L27dszxhiLj49nANjRo0cV1kF6jmPHjsm2HTp0iAFgL168UMt1EkJqh1pCCCEad//+fRQWFuL111+Hubm57LZt2zY8ePBAVi4wMFD2u42NDby8vBAXFwcAiIuLQ8+ePeWO27NnT9y7dw9isRjXrl2DUChEnz59qq1Lx44dZb87OzsDANLT0+t9jYSQ2jPQdQUIIY1ffn4+AODQoUNwdXWVe0wkEskFInVlYmKiUjlDQ0PZ7xzHAeDHqxBCtI9aQgghGte+fXuIRCIkJiaidevWcjd3d3dZuXPnzsl+z87Oxt27d+Ht7Q0A8Pb2xpkzZ+SOe+bMGbRt2xZCoRC+vr6QSCRyY0wIIfqNWkIIIRpnYWGBjz/+GB9++CEkEgn+85//IDc3F2fOnIGlpSU8PDwAAIsXL4atrS0cHR3x2Wefwc7ODiEhIQCAjz76CF27dsWSJUsQFhaGmJgYrFu3Dt999x0AwNPTE5MmTcLUqVOxZs0a+Pn54fHjx0hPT8fo0aN1demEkGpQEEII0YolS5bA3t4ekZGRePjwIaytrdGpUyfMnz9f1h3y9ddfY/bs2bh37x78/f3xxx9/wMjICADQqVMn7N69GwsWLMCSJUvg7OyMxYsXY/LkybJzbNiwAfPnz8f777+PrKwsNG/eHPPnz9fF5RJCVMAxxpiuK0EIadqio6PRt29fZGdnw9raWtfVIYRoCY0JIYQQQohOUBBCCCGEEJ2g7hhCCCGE6AS1hBBCCCFEJygIIYQQQohOUBBCCCGEEJ2gIIQQQgghOkFBCCGEEEJ0goIQQgghhOgEBSGEEEII0QkKQgghhBCiE/8PPCrUYIsIwkoAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 600x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* background: */\n",
       "    progress::-webkit-progress-bar {background-color: #CDCDCD; width: 100%;}\n",
       "    progress {background-color: #CDCDCD;}\n",
       "\n",
       "    /* value: */\n",
       "    progress::-webkit-progress-value {background-color: #00BFFF  !important;}\n",
       "    progress::-moz-progress-bar {background-color: #00BFFF  !important;}\n",
       "    progress {color: #00BFFF ;}\n",
       "\n",
       "    /* optional */\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #000000;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      <progress value='20' class='progress-bar-interrupted' max='100' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      20.00% [20/100 1:17:04<5:08:17][earlystopping]\n",
       "      <br>\n",
       "      \n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< val_loss without improvement in 10 epoch,early stopping >>>>>>\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>lr</th>\n",
       "      <th>val_loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.085002</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.096762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.069643</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.090588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.068870</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.086552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.064769</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.083662</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.065533</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.079649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>0.065219</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.081573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>0.065145</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.080994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>0.067934</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.076796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>0.058139</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.076972</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>0.065717</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.074264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>0.054295</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.088025</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>0.050960</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.078092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>0.053583</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.075034</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>0.049259</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.076480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>0.038436</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.081660</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>16</td>\n",
       "      <td>0.043832</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.079644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>17</td>\n",
       "      <td>0.038264</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.084635</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>18</td>\n",
       "      <td>0.042769</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.098947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>19</td>\n",
       "      <td>0.036059</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.088988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>20</td>\n",
       "      <td>0.027864</td>\n",
       "      <td>0.00006</td>\n",
       "      <td>0.106413</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    epoch  train_loss       lr  val_loss\n",
       "0       1    0.085002  0.00006  0.096762\n",
       "1       2    0.069643  0.00006  0.090588\n",
       "2       3    0.068870  0.00006  0.086552\n",
       "3       4    0.064769  0.00006  0.083662\n",
       "4       5    0.065533  0.00006  0.079649\n",
       "5       6    0.065219  0.00006  0.081573\n",
       "6       7    0.065145  0.00006  0.080994\n",
       "7       8    0.067934  0.00006  0.076796\n",
       "8       9    0.058139  0.00006  0.076972\n",
       "9      10    0.065717  0.00006  0.074264\n",
       "10     11    0.054295  0.00006  0.088025\n",
       "11     12    0.050960  0.00006  0.078092\n",
       "12     13    0.053583  0.00006  0.075034\n",
       "13     14    0.049259  0.00006  0.076480\n",
       "14     15    0.038436  0.00006  0.081660\n",
       "15     16    0.043832  0.00006  0.079644\n",
       "16     17    0.038264  0.00006  0.084635\n",
       "17     18    0.042769  0.00006  0.098947\n",
       "18     19    0.036059  0.00006  0.088988\n",
       "19     20    0.027864  0.00006  0.106413"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# keras_model.load_ckpt(ckpt_path) #支持加载微调后的权重继续训练(断点续训)\n",
    "keras_model.fit(train_data = dl_train,\n",
    "                val_data = dl_val,\n",
    "                epochs=100,patience=10,\n",
    "                monitor='val_loss',mode='min',\n",
    "                ckpt_path = ckpt_path\n",
    "               )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "108976a7-17cc-411e-8532-d781d65d1f29",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82a55a90-f980-4dbd-b8f8-6c0e0c8d5a5b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "455498a9-3613-4efa-b7d4-6035314c8fdc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "49763c6a-6ee5-448c-b873-48825a6fba67",
   "metadata": {},
   "source": [
    "## 四，保存模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f98c4e5-dbb0-456e-bc4b-1c8dfccbd499",
   "metadata": {},
   "source": [
    "为减少GPU压力，此处可重启kernel释放显存"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ee686c58-42e2-463a-a429-3fe16dee5a7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings \n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "142a917e-9db0-4d5a-8352-cb250561f47d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-07-19 23:21:49,363] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The model weights are not tied. Please use the `tie_weights` method before using the `infer_auto_device` function.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "23c6bf03a09d40838ad3c888d990e49e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, AutoModel, BitsAndBytesConfig\n",
    "from transformers.generation.utils import GenerationConfig\n",
    "import torch.nn as nn\n",
    "model_name_or_path ='../baichuan-13b'\n",
    "ckpt_path = 'baichuan13b_waimai'\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "    model_name_or_path,\n",
    "    trust_remote_code=True\n",
    ")\n",
    "model_old = AutoModelForCausalLM.from_pretrained(\n",
    "    model_name_or_path,\n",
    "    trust_remote_code=True,\n",
    "    low_cpu_mem_usage=True,\n",
    "    torch_dtype=torch.float16,\n",
    "    device_map='auto'\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2a8b409f-f100-48c5-9794-62cb5bdf19d4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===================================BUG REPORT===================================\n",
      "Welcome to bitsandbytes. For bug reports, please run\n",
      "\n",
      "python -m bitsandbytes\n",
      "\n",
      " and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
      "================================================================================\n",
      "bin /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so\n",
      "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...\n",
      "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0\n",
      "CUDA SETUP: Highest compute capability among GPUs detected: 8.0\n",
      "CUDA SETUP: Detected CUDA version 116\n",
      "CUDA SETUP: Loading binary /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so...\n"
     ]
    }
   ],
   "source": [
    "from peft import PeftModel\n",
    "\n",
    "#可能需要5分钟左右\n",
    "peft_model = PeftModel.from_pretrained(model_old, ckpt_path)\n",
    "model_new = peft_model.merge_and_unload()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2c060b4a-7731-4ec7-8699-9f47088f4a2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers.generation.utils import GenerationConfig\n",
    "model_new.generation_config = GenerationConfig.from_pretrained(model_name_or_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca83a654-1d18-47e7-bbf1-d67ca351813d",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "524a3d55-5daf-4ff1-8fb9-e3dd60d65f18",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "乔戈里峰。世界第二高峰———乔戈里峰西方登山者称其为k2峰，海拔高度是8611米，位于喀喇昆仑山脉的中巴边境上\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import clear_output\n",
    "messages = []\n",
    "messages.append({\"role\": \"user\",\n",
    "                 \"content\": \"世界上第二高的山峰是什么？\"})\n",
    "response = model_new.chat(tokenizer,messages=messages,stream=True)\n",
    "for res in response:\n",
    "    print(res)\n",
    "    clear_output(wait=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ce165cd8-987d-47be-95ac-83cf7ef479b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "save_path = 'baichuan-13b-waimai'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9214d354-811b-4e47-96ac-682ed5e68ee9",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.save_pretrained(save_path)\n",
    "model_new.save_pretrained(save_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a7950603-e844-4d60-990d-f86245fbaa26",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp ../baichuan-13b/*.py  baichuan-13b-waimai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a91c249-3402-489b-ae8e-5554a5082562",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "81582c3e-28d3-4847-a84c-dc1b53aa0839",
   "metadata": {},
   "source": [
    "## 五，使用模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52c57c2f-0e1a-4e77-b4e4-fefa210b1bb1",
   "metadata": {},
   "source": [
    "为减少GPU压力，此处可再次重启kernel释放显存。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2bec4773-61f0-4aa2-9ac2-d2e1c295b2e3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dc27e2a0c9414f79a4da3b8466dce922",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, BitsAndBytesConfig\n",
    "from transformers.generation.utils import GenerationConfig\n",
    "import torch.nn as nn\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "model_name_or_path = 'baichuan-13b-waimai'\n",
    "\n",
    "bnb_config=BitsAndBytesConfig(\n",
    "            load_in_4bit=True,\n",
    "            bnb_4bit_compute_dtype=torch.float16,\n",
    "            bnb_4bit_use_double_quant=True,\n",
    "            bnb_4bit_quant_type=\"nf4\",\n",
    "            llm_int8_threshold=6.0,\n",
    "            llm_int8_has_fp16_weight=False,\n",
    "        )\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "   model_name_or_path, trust_remote_code=True)\n",
    "\n",
    "model = AutoModelForCausalLM.from_pretrained(model_name_or_path,\n",
    "                quantization_config=bnb_config,\n",
    "                trust_remote_code=True) \n",
    "\n",
    "model.generation_config = GenerationConfig.from_pretrained(model_name_or_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e958b920-63cd-4e1a-bd28-d36e7c281901",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "乔戈里峰。世界第二高峰———乔戈里峰\n",
      "海拔高度：8610米\n",
      "坐标纬度：35°49′15′′n,76°21′24′′e\n",
      "地理位置：喀喇昆仑山脉中巴边境上\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import clear_output\n",
    "messages = []\n",
    "messages.append({\"role\": \"user\",\n",
    "                 \"content\": \"世界上第二高的山峰是什么？\"})\n",
    "response = model.chat(tokenizer,messages=messages,stream=True)\n",
    "for res in response:\n",
    "    print(res)\n",
    "    clear_output(wait=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e63d6988-4cfd-46ba-8a5d-f11e5d183483",
   "metadata": {},
   "source": [
    "我们测试一下微调后的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b8e7ad1b-9831-4b11-9ef8-a848a49c9882",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np \n",
    "import datasets \n",
    "from tqdm import tqdm \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "57a77d28-11ab-423c-8856-c4027d2e9faf",
   "metadata": {},
   "outputs": [],
   "source": [
    "prefix = \"\"\"外卖评论文本分类任务:\n",
    "下面是一些范例:\n",
    "\n",
    "味道真不错 -> 好评\n",
    "太辣了，吃不下都  -> 差评\n",
    "\n",
    "请对下述评论进行分类。返回'好评'或者'差评'。\n",
    "\"\"\"\n",
    "\n",
    "def get_prompt(text):\n",
    "    return prefix+text+' -> '\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "465724e3-70ef-4500-9154-eaf626f7701e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "好评\n"
     ]
    }
   ],
   "source": [
    "messages  = [{\"role\": \"user\", \"content\": get_prompt('味道不错，下次再来')}]\n",
    "response = model.chat(tokenizer, messages)\n",
    "print(response)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d4130efa-795f-4d2a-ac96-378040ebc540",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'role': 'user', 'content': \"外卖评论文本分类任务:\\n下面是一些范例:\\n\\n味道真不错 -> 好评\\n太辣了，吃不下都  -> 差评\\n\\n请对下述评论进行分类。返回'好评'或者'差评'。\\n味道不错，下次再来 -> \"}, {'role': 'assistant', 'content': '好评'}]\n"
     ]
    }
   ],
   "source": [
    "messages = messages+[{\"role\": \"assistant\", \"content\": response}]\n",
    "print(messages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "50bad51c-2f1d-4af2-bb63-42cf845bf802",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_message(prompt,response):\n",
    "    return [{\"role\": \"user\", \"content\": f'{prompt} -> '},\n",
    "            {\"role\": \"assistant\", \"content\": response}]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "89f4cd73-b6a9-47fe-aae8-b8d5b66a2d30",
   "metadata": {},
   "outputs": [],
   "source": [
    "messages.extend(get_message('太贵了','差评'))\n",
    "messages.extend(get_message('非常快，味道好','好评'))\n",
    "messages.extend(get_message('这么咸，真的是醉了','差评'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "c8c57fea-721e-45d3-81b5-3b5500d21b7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(text,temperature=0.01):\n",
    "    model.generation_config.temperature=temperature\n",
    "    response = model.chat(tokenizer, \n",
    "                          messages = messages+[{'role':'user','content':f'{text} -> '}])\n",
    "    return response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "aed22302-e528-4563-bd07-7c5fcd8c0dc5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'好评'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict('死鬼，怎么弄得这么好吃呢')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "db357a8c-83d4-4de4-83b9-f1cb0e284846",
   "metadata": {},
   "outputs": [],
   "source": [
    "#数据集加载\n",
    "dftest = pd.read_parquet('../data/dftest.parquet')[['text','label','tag']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "44e765ac-21be-475d-9076-32ddcc867df3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2000/2000 [10:24<00:00,  3.20it/s]\n"
     ]
    }
   ],
   "source": [
    "dftest['pred'] = [predict(text) for text in tqdm(dftest['text'])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "bdc2832d-3f0d-49b2-93e9-d307ce771f1b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc =  0.9015\n"
     ]
    }
   ],
   "source": [
    "print('acc = ',len(dftest.query('tag==pred'))/len(dftest))  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e61d9def-fb3b-4bf1-8579-b5e28415082c",
   "metadata": {},
   "source": [
    "微调后的acc为0.9015，相比微调前的0.8925，约提升1个百分点。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c81a304e-2e8b-4fe2-bc79-e2ed9bbadd5c",
   "metadata": {},
   "source": [
    "**如果本项目对你有所帮助，想鼓励一下作者，记得给本项目加一颗星星star⭐️，并分享给你的朋友们喔😊!** \n",
    "\n",
    "如果在torchkeras的使用中遇到问题，可以在项目中提交issue。\n",
    "\n",
    "如果想要获得更快的反馈或者与其他torchkeras用户小伙伴进行交流，\n",
    "\n",
    "可以在公众号算法美食屋后台回复关键字：**加群**。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h41m2zugguj20k00b9q46.jpg)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}