README.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building an LLM Router for High-Quality and Cost-Effective Responses\n",
    "\n",
    "## TLDR\n",
    "1. We introduce a framework for training state-of-the-art *LLM routers*, systems that dynamically direct queries to either high-quality closed LLMs or cost-effective open-source LLMs, based on query complexity, optimizing both response quality and cost.\n",
    "\n",
    "2. This tutorial provides an in-depth guide on building an LLM router *based on a causal-LLM classifier*, starting with generating labeled data, finetuning an LLM-based classifier with Anyscale's API, and finally running offline evaluations.\n",
    "\n",
    "3. In collaboration with the Berkeley LMSys group, we release an [arXiv paper](https://arxiv.org/pdf/2406.18665) presenting extensive evaluations of this model along with other models. Overall, our LLM Routers can achieve the same performance as our baselines with up to a 70% cost reduction on MT Bench, a 30% cost reduction on MMLU, and a 40% cost reduction on GSM8K.\n",
    "\n",
    "# Background\n",
    "When developing applications using Large Language Models (LLMs), achieving high-quality responses while maintaining a budget is a key challenge. Closed models like GPT-4 provide superior quality but are costly, especially with a high volume of queries. Conversely, Open Source Software (OSS) models are more economical but may not match the quality, especially for complex or domain-specific queries.\n",
    "\n",
    "An **LLM Router** helps balance these aspects by deciding which queries are routed to a closed LLM and which to an OSS LLM based on the query's complexity or domain specificity. Below is a schematic representation of an LLM Router:\n",
    "\n",
    "<div style=\"text-align: center;\">\n",
    "    <img src=\"assets/llm-router-flowchart_2.png\" alt=\"LLM Router\" width=\"800\"/>\n",
    "</div>\n",
    "\n",
    "Given a set of user queries, an LLM router enables generating high-quality LLM responses while minimizing the overall cost.\n",
    "\n",
    "# Approach\n",
    "\n",
    "In this tutorial, we'll demonstrate how to train a *causal-LLM classifier* on the Anyscale platform as an effective LLM router. We make the following design choices:\n",
    "\n",
    "1. **Model Choices**: We’ll use GPT-4 as an example of a closed LLM and Mixtral-8x7B as the OSS LLM, so our causal LLM classifier will route between these two models.\n",
    "2. **Response Quality Rating**: We'll quantify the quality of an LLM response on a scale of 1 to 5 stars, with higher scores indicating better quality. For simplicity, we'll assume that GPT-4 always achieves a 5-star rating, so it serves as a reference for Mixtral-8x7B.\n",
    "3. **Causal LLM Classifier**: We'll finetune a Llama3-8B model as our causal LLM classifier and leverage Anyscale's powerful API. [Our research](https://arxiv.org/pdf/2406.18665) shows that this model offers superior routing performance compared to smaller architectures.\n",
    "\n",
    "More concretely, the objective of the causal LLM classifier is to direct \"simple\" queries to Mixtral-8x7B, thereby maintaining high overall response quality (e.g., an average score of 4.8/5) while significantly reducing costs (e.g., by 50%).\n",
    "\n",
    "We show that it's possible to build LLM routers that achieve outstanding performance. Below are results from our best-performing LLM routers, the Causal LLM and a Matrix Factorization (MF) model, evaluated on the [MT Bench benchmark](https://arxiv.org/pdf/2306.05685), which demonstrate that our routers can achieve higher quality with lower costs (i.e., fewer calls to GPT-4) compared to the random baseline and public LLM routing systems from Unify AI and Martian. For more details on these results and additional ones, refer to our paper.\n",
    "\n",
    "<div style=\"display: flex; justify-content: center; align-items: center; gap: 20px;\">\n",
    "    <img src=\"assets/indep-benchmarks.png\" alt=\"Benchmark 1\" style=\"width: 400px;\"/>\n",
    "    <img src=\"assets/indep-benchmarks-llama.png\" alt=\"Benchmark 2\" style=\"width: 400px;\"/>\n",
    "</div>\n",
    "\n",
    "In the following sections, we discuss the steps that enable anyone to build a strong LLM router.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "\n",
    "1. [**Prepare Labeled Data**](#generate-labeled-data): The foundation of a robust LLM router is high-quality labeled data. In this section, we'll guide you through preparing this training data.\n",
    "\n",
    "2. [**Finetune a Router Model**](#finetune-router-model): We demonstrate how to finetune a causal-LLM classifier using Anyscale's finetuning API, transforming it into an effective LLM router.\n",
    "\n",
    "3. [**Offline Evaluation**](#offline-eval): Using the public codebase ([RouteLLM](https://github.com/lm-sys/RouteLLM)), we will walk through an offline evaluation on standard benchmarks.\n",
    "\n",
    "**Time to complete**: Approximately 120 minutes, including time to train on a node with 8xA10 GPUs.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install required packages\n",
    "!pip install -r requirements.txt\n",
    "\n",
    "# Store your ANYSCALE_API_KEY and OPENAI_API_KEY in /home/ray/default/.env\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"/home/ray/default/.env\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 1: Prepare Labeled Data <a id=\"generate-labeled-data\"></a>\n",
    "\n",
    "The llm router essentially functions as a binary classifier, deciding whether to route a query to GPT-4 or Mixtral-8x7B based on the query text. Initially, we considered labeled data in the format `(query, routing_label)`, where `routing_label` is 1 if the query should be routed to Mixtral-8x7B and 0 if it should be routed to GPT-4.\n",
    "\n",
    "However, our early experiments revealed that *binary labels do not provide sufficient signal for training a robust router model*. Therefore, we adopted a different labeling approach using a *1-5 scoring system*, which reflects how well Mixtral-8x7B can effectively respond to the user's query. More specifically:\n",
    "\n",
    "- **4-5**: Mixtral-8x7B produces a very strong answer, showing deep understanding, creativity, detailed insight, and high relevance.\n",
    "- **3**: Mixtral-8x7B provides an adequate answer with moderate detail, relevance, and factual accuracy.\n",
    "- **1-2**: Mixtral-8x7B struggles to produce a strong answer due to the question's difficulty, vagueness, or the model's limitations.\n",
    "\n",
    "We use labeled samples in the format `(query, score_label)`. The `routing_label` can be derived from the `score_label` by setting a score threshold for quality, i.e. `routing_label = 1 if score_label >= 4 else 0`.\n",
    "\n",
    "Next, we'll dive into the detailed process of preparing our labeled dataset.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1: Query Dataset\n",
    "\n",
    "We want our llm router to be effective in open-ended chat domains. So, our first step is to collect a set of generic queries from the [Nectar dataset](https://huggingface.co/datasets/berkeley-nest/Nectar). We chose the Nectar dataset for two reasons: it combines queries from many different domains, including open-ended chat, and it has responses from many models, including over 191K responses from GPT-4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>answers</th>\n",
       "      <th>turns</th>\n",
       "      <th>num_responses</th>\n",
       "      <th>source</th>\n",
       "      <th>good_natured</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\n\\nHuman: 0.002 = 1000 \\n1 = x?\\n\\nAssistant:</td>\n",
       "      <td>[{'answer': 'To find the value of x, we can set up a proportion using the given information:\n",
       "\n",
       "0.002/1000 = 1/x\n",
       "\n",
       "To solve for x, we can cross multiply:\n",
       "\n",
       "0.002 * x = 1000 * 1\n",
       "\n",
       "0.002x = 1000\n",
       "\n",
       "Dividing both sides by 0.002:\n",
       "\n",
       "x = 1000 / 0.002\n",
       "\n",
       "x = 500,000\n",
       "\n",
       "Therefore, 1 is equal to 500,000 in this proportion.', 'model': 'gpt-3.5-turbo', 'rank': 1.0}, {'answer': 'To solve for x in this equation, you can use cross-multiplication. Here's how:\n",
       "\n",
       "0.002 / 1 = 1000 / x\n",
       "\n",
       "Cross-multiplication gives:\n",
       "\n",
       "0.002 * x = 1000\n",
       "\n",
       "To solve for x, divide both sides by 0.002:\n",
       "\n",
       "x = 1000 / 0.002\n",
       "\n",
       "So, x equals 500,000.', 'model': 'gpt-4-0613', 'rank': 2.0}, {'answer': 'This looks like a proportion. To solve for x, you can set up a proportion equation:\n",
       "\n",
       "0.002 / 1 = 1000 / x\n",
       "\n",
       "Now, cross-multiply:\n",
       "\n",
       "0.002 \\* x = 1 \\* 1000\n",
       "\n",
       "Then, solve for x:\n",
       "\n",
       "0.002x = 1000\n",
       "\n",
       "x = 1000 / 0.002\n",
       "\n",
       "x = 500,000', 'model': 'gpt-3.5-turbo', 'rank': 3.0}, {'answer': 'If 0.002 equals 1000, then 1 would equal 500,000.', 'model': 'gpt-4', 'rank': 4.0}, {'answer': 'I apologize, but the equation you provided is not correct.\n",
       "\n",
       "0.002 is equal to 0.002, not 1000.\n",
       "\n",
       "1 is equal to x, not a number.\n",
       "\n",
       "Can you please provide a correct equation or problem for me to solve?', 'model': 'llama-2-7b-chat', 'rank': 5.0}, {'answer': '0.001 = x', 'model': 'gpt-3.5-turbo-instruct', 'rank': 6.0}, {'answer': 'It seems like you are asking for the value of x in the equation x = 1/0.002. \n",
       "\n",
       "To solve this equation, you can divide both sides by 0.002 to get: \n",
       "\n",
       "x = 1/0.002 \n",
       "x = 500 \n",
       "\n",
       "Therefore, x = 500.', 'model': 'mistral-7b-instruct-v0.1', 'rank': 7.0}]</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>[sharegpt]</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            prompt  \\\n",
       "0  \\n\\nHuman: 0.002 = 1000 \\n1 = x?\\n\\nAssistant:    \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         answers  \\\n",
       "0  [{'answer': 'To find the value of x, we can set up a proportion using the given information:\n",
       "\n",
       "0.002/1000 = 1/x\n",
       "\n",
       "To solve for x, we can cross multiply:\n",
       "\n",
       "0.002 * x = 1000 * 1\n",
       "\n",
       "0.002x = 1000\n",
       "\n",
       "Dividing both sides by 0.002:\n",
       "\n",
       "x = 1000 / 0.002\n",
       "\n",
       "x = 500,000\n",
       "\n",
       "Therefore, 1 is equal to 500,000 in this proportion.', 'model': 'gpt-3.5-turbo', 'rank': 1.0}, {'answer': 'To solve for x in this equation, you can use cross-multiplication. Here's how:\n",
       "\n",
       "0.002 / 1 = 1000 / x\n",
       "\n",
       "Cross-multiplication gives:\n",
       "\n",
       "0.002 * x = 1000\n",
       "\n",
       "To solve for x, divide both sides by 0.002:\n",
       "\n",
       "x = 1000 / 0.002\n",
       "\n",
       "So, x equals 500,000.', 'model': 'gpt-4-0613', 'rank': 2.0}, {'answer': 'This looks like a proportion. To solve for x, you can set up a proportion equation:\n",
       "\n",
       "0.002 / 1 = 1000 / x\n",
       "\n",
       "Now, cross-multiply:\n",
       "\n",
       "0.002 \\* x = 1 \\* 1000\n",
       "\n",
       "Then, solve for x:\n",
       "\n",
       "0.002x = 1000\n",
       "\n",
       "x = 1000 / 0.002\n",
       "\n",
       "x = 500,000', 'model': 'gpt-3.5-turbo', 'rank': 3.0}, {'answer': 'If 0.002 equals 1000, then 1 would equal 500,000.', 'model': 'gpt-4', 'rank': 4.0}, {'answer': 'I apologize, but the equation you provided is not correct.\n",
       "\n",
       "0.002 is equal to 0.002, not 1000.\n",
       "\n",
       "1 is equal to x, not a number.\n",
       "\n",
       "Can you please provide a correct equation or problem for me to solve?', 'model': 'llama-2-7b-chat', 'rank': 5.0}, {'answer': '0.001 = x', 'model': 'gpt-3.5-turbo-instruct', 'rank': 6.0}, {'answer': 'It seems like you are asking for the value of x in the equation x = 1/0.002. \n",
       "\n",
       "To solve this equation, you can divide both sides by 0.002 to get: \n",
       "\n",
       "x = 1/0.002 \n",
       "x = 500 \n",
       "\n",
       "Therefore, x = 500.', 'model': 'mistral-7b-instruct-v0.1', 'rank': 7.0}]   \n",
       "\n",
       "   turns  num_responses      source  good_natured  \n",
       "0      1              7  [sharegpt]          True  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of queries with GPT-4 responses: 191487\n"
     ]
    }
   ],
   "source": [
    "from src.utils import load_and_display_nectar\n",
    "\n",
    "nectar_df = load_and_display_nectar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Data Preprocessing\n",
    "\n",
    "We will use a subset of the Nectar data that includes responses from GPT-4, as these will be used to generate scores (as seen below). We will process this data by focusing on single-turn conversations, filtering for good-natured interactions, and cleaning up the prompts and responses to maintain high quality. Additionally, we will sample a small subset from the dataset for the purpose of this tutorial; however, you can skip sampling to work with the full dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.utils import preprocess_nectar\n",
    "\n",
    "nectar_gpt4_df = preprocess_nectar(\n",
    "    nectar_df, model=\"gpt-4\", response_column=\"gpt4_response\"\n",
    ")\n",
    "\n",
    "# Sample a small subset from the dataset for the purpose of this tutorial\n",
    "N_SUBSET = 30\n",
    "dataset_df = nectar_gpt4_df.sample(N_SUBSET, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset overview with GPT-4 responses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>source</th>\n",
       "      <th>gpt4_response</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6062</th>\n",
       "      <td>Based on the features mentioned, which hotel d...</td>\n",
       "      <td>[evol_instruct]</td>\n",
       "      <td>Based on the features mentioned, Hotel A seems...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>113830</th>\n",
       "      <td>Provide step-by-step instructions on how to cr...</td>\n",
       "      <td>[ultrachat]</td>\n",
       "      <td>Sure, here's a simple step-by-step guide on ho...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138869</th>\n",
       "      <td>What are the 10 largest cities in the US by po...</td>\n",
       "      <td>[lmsys-chat-1m]</td>\n",
       "      <td>As of the most recent data available, the 10 l...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169249</th>\n",
       "      <td>Write a comparison essay of at least 500 words...</td>\n",
       "      <td>[ultrachat]</td>\n",
       "      <td>Title: A Comparative Analysis of Driving a Car...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116934</th>\n",
       "      <td>Q: You are provided with an \"Event\", \"Intent\" ...</td>\n",
       "      <td>[flan_v2_niv2]</td>\n",
       "      <td>PersonX might feel satisfied or content using ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   prompt           source  \\\n",
       "6062    Based on the features mentioned, which hotel d...  [evol_instruct]   \n",
       "113830  Provide step-by-step instructions on how to cr...      [ultrachat]   \n",
       "138869  What are the 10 largest cities in the US by po...  [lmsys-chat-1m]   \n",
       "169249  Write a comparison essay of at least 500 words...      [ultrachat]   \n",
       "116934  Q: You are provided with an \"Event\", \"Intent\" ...   [flan_v2_niv2]   \n",
       "\n",
       "                                            gpt4_response  \n",
       "6062    Based on the features mentioned, Hotel A seems...  \n",
       "113830  Sure, here's a simple step-by-step guide on ho...  \n",
       "138869  As of the most recent data available, the 10 l...  \n",
       "169249  Title: A Comparative Analysis of Driving a Car...  \n",
       "116934  PersonX might feel satisfied or content using ...  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(dataset_df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Data Labeling\n",
    "\n",
    "We don't have human labels for scores, so we will use the [LLM-as-a-Judge approach](https://arxiv.org/abs/2306.05685). GPT-4 will act as an evaluator, reviewing the query and Mixtral's response to provide a score from 1-5. As shown in the paper, the most robust way to get labels is by providing a reference answer for comparison. Here, GPT-4's own response serves as the reference, and Mixtral's response is evaluated against it.\n",
    "\n",
    "There are two main steps in this process:\n",
    "1. **Generate Mixtral-8x7B responses for all queries**: We will use an online batch-inference method utilizing Ray and Anyscale endpoints.\n",
    "2. **Generate LLM-as-a-Judge labels**: We will ask GPT-4 to evaluate the Mixtral responses against its own reference answers and provide a score from 1-5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate Mixtral-8x7B Responses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting batch inference on 30 queries...\n",
      "# queries un-processed: 29, in-progress: 1, ready: 0\n",
      "# queries un-processed: 28, in-progress: 2, ready: 0\n",
      "# queries un-processed: 27, in-progress: 2, ready: 1\n",
      "# queries un-processed: 26, in-progress: 3, ready: 0\n",
      "# queries un-processed: 25, in-progress: 4, ready: 0\n",
      "# queries un-processed: 24, in-progress: 5, ready: 0\n",
      "# queries un-processed: 23, in-progress: 6, ready: 0\n",
      "# queries un-processed: 22, in-progress: 7, ready: 0\n",
      "# queries un-processed: 21, in-progress: 8, ready: 0\n",
      "# queries un-processed: 20, in-progress: 9, ready: 0\n",
      "# queries un-processed: 19, in-progress: 10, ready: 0\n",
      "# queries un-processed: 18, in-progress: 10, ready: 1\n",
      "# queries un-processed: 17, in-progress: 11, ready: 0\n",
      "# queries un-processed: 16, in-progress: 12, ready: 0\n",
      "# queries un-processed: 15, in-progress: 13, ready: 0\n",
      "# queries un-processed: 14, in-progress: 14, ready: 0\n",
      "# queries un-processed: 13, in-progress: 14, ready: 1\n",
      "# queries un-processed: 12, in-progress: 14, ready: 1\n",
      "# queries un-processed: 11, in-progress: 14, ready: 1\n",
      "# queries un-processed: 10, in-progress: 15, ready: 0\n",
      "# queries un-processed: 9, in-progress: 15, ready: 1\n",
      "# queries un-processed: 8, in-progress: 15, ready: 1\n",
      "# queries un-processed: 7, in-progress: 15, ready: 1\n",
      "# queries un-processed: 6, in-progress: 15, ready: 1\n",
      "# queries un-processed: 5, in-progress: 16, ready: 0\n",
      "# queries un-processed: 4, in-progress: 16, ready: 1\n",
      "# queries un-processed: 3, in-progress: 17, ready: 0\n",
      "# queries un-processed: 2, in-progress: 18, ready: 0\n",
      "# queries un-processed: 1, in-progress: 18, ready: 1\n",
      "# queries un-processed: 0, in-progress: 18, ready: 1\n",
      "# queries un-processed: 0, in-progress: 17, ready: 1\n",
      "# queries un-processed: 0, in-progress: 17, ready: 0\n",
      "# queries un-processed: 0, in-progress: 16, ready: 1\n",
      "# queries un-processed: 0, in-progress: 16, ready: 0\n",
      "# queries un-processed: 0, in-progress: 15, ready: 1\n",
      "# queries un-processed: 0, in-progress: 14, ready: 1\n",
      "# queries un-processed: 0, in-progress: 13, ready: 1\n",
      "# queries un-processed: 0, in-progress: 12, ready: 1\n",
      "# queries un-processed: 0, in-progress: 11, ready: 1\n",
      "# queries un-processed: 0, in-progress: 10, ready: 1\n",
      "# queries un-processed: 0, in-progress: 9, ready: 1\n",
      "# queries un-processed: 0, in-progress: 9, ready: 0\n",
      "# queries un-processed: 0, in-progress: 8, ready: 1\n",
      "# queries un-processed: 0, in-progress: 8, ready: 0\n",
      "# queries un-processed: 0, in-progress: 7, ready: 1\n",
      "# queries un-processed: 0, in-progress: 6, ready: 1\n",
      "# queries un-processed: 0, in-progress: 6, ready: 0\n",
      "# queries un-processed: 0, in-progress: 5, ready: 1\n",
      "# queries un-processed: 0, in-progress: 5, ready: 0\n",
      "# queries un-processed: 0, in-progress: 5, ready: 0\n",
      "# queries un-processed: 0, in-progress: 5, ready: 0\n",
      "# queries un-processed: 0, in-progress: 4, ready: 1\n",
      "# queries un-processed: 0, in-progress: 3, ready: 1\n",
      "# queries un-processed: 0, in-progress: 2, ready: 1\n",
      "# queries un-processed: 0, in-progress: 2, ready: 0\n",
      "# queries un-processed: 0, in-progress: 1, ready: 1\n",
      "# queries un-processed: 0, in-progress: 1, ready: 0\n",
      "# queries un-processed: 0, in-progress: 0, ready: 1\n",
      "Done in 19.21sec.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from src.online_inference import generate_mixtral_responses\n",
    "\n",
    "dataset_df = generate_mixtral_responses(\n",
    "    dataset_df, os.getenv(\"ANYSCALE_API_KEY\"), response_column=\"mixtral_response\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset overview with Mixtral responses\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>source</th>\n",
       "      <th>gpt4_response</th>\n",
       "      <th>mixtral_response</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6062</th>\n",
       "      <td>Based on the features mentioned, which hotel d...</td>\n",
       "      <td>[evol_instruct]</td>\n",
       "      <td>Based on the features mentioned, Hotel A seems...</td>\n",
       "      <td>Based on the information provided, I would sa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>113830</th>\n",
       "      <td>Provide step-by-step instructions on how to cr...</td>\n",
       "      <td>[ultrachat]</td>\n",
       "      <td>Sure, here's a simple step-by-step guide on ho...</td>\n",
       "      <td>Sure, I'd be happy to help you make a homemad...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138869</th>\n",
       "      <td>What are the 10 largest cities in the US by po...</td>\n",
       "      <td>[lmsys-chat-1m]</td>\n",
       "      <td>As of the most recent data available, the 10 l...</td>\n",
       "      <td>Sure, I'd be happy to help with that! Here ar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169249</th>\n",
       "      <td>Write a comparison essay of at least 500 words...</td>\n",
       "      <td>[ultrachat]</td>\n",
       "      <td>Title: A Comparative Analysis of Driving a Car...</td>\n",
       "      <td>Title: The Great Debate: Driving a Car vs. Ri...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116934</th>\n",
       "      <td>Q: You are provided with an \"Event\", \"Intent\" ...</td>\n",
       "      <td>[flan_v2_niv2]</td>\n",
       "      <td>PersonX might feel satisfied or content using ...</td>\n",
       "      <td>PersonX probably feels comfortable and focuse...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   prompt           source  \\\n",
       "6062    Based on the features mentioned, which hotel d...  [evol_instruct]   \n",
       "113830  Provide step-by-step instructions on how to cr...      [ultrachat]   \n",
       "138869  What are the 10 largest cities in the US by po...  [lmsys-chat-1m]   \n",
       "169249  Write a comparison essay of at least 500 words...      [ultrachat]   \n",
       "116934  Q: You are provided with an \"Event\", \"Intent\" ...   [flan_v2_niv2]   \n",
       "\n",
       "                                            gpt4_response  \\\n",
       "6062    Based on the features mentioned, Hotel A seems...   \n",
       "113830  Sure, here's a simple step-by-step guide on ho...   \n",
       "138869  As of the most recent data available, the 10 l...   \n",
       "169249  Title: A Comparative Analysis of Driving a Car...   \n",
       "116934  PersonX might feel satisfied or content using ...   \n",
       "\n",
       "                                         mixtral_response  \n",
       "6062     Based on the information provided, I would sa...  \n",
       "113830   Sure, I'd be happy to help you make a homemad...  \n",
       "138869   Sure, I'd be happy to help with that! Here ar...  \n",
       "169249   Title: The Great Debate: Driving a Car vs. Ri...  \n",
       "116934   PersonX probably feels comfortable and focuse...  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(dataset_df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate GPT-4-as-a-judge scores "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first take a look at an example query we will send to GPT-4 for judgement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Instruction]\n",
      "Evaluate the AI assistant's proficiency in answering the user question displayed below. Your evaluation should consider factors such as the helpfulness, relevance, adherence to real-world facts, depth, creativity, and level of detail of the response. You will be given a reference answer which is considered of high quality. Your assessment will have two lines: First line has a rating on a scale of 1 to 5 with a higher rating representing higher response quality. Follow strictly this format: \"[[rating]]\", for example: \"[[3]]\". Second line contains a short explanation of your rating.\n",
      "\n",
      "[Question]\n",
      "Q: You are provided with an \"Event\", \"Intent\" related to PersonX. Guess a reaction/reaction of PersonX about the given event and their intention.\n",
      "Event:PersonX uses ___ in class. Intent: 1) to use his prefered writing implement\n",
      "A:\n",
      "\n",
      "[Reference Answer]\n",
      "PersonX might feel satisfied or content using their preferred writing implement in class, as it aligns with their intention to utilize a comfortable and desired tool for writing. \n",
      "Confidence: 85%\n",
      "\n",
      "[Assistant Answer]\n",
      " PersonX probably feels comfortable and focused in class, as they are using their preferred writing implement. This may help them engage more effectively with the material being taught.\n",
      "\n",
      "Guidelines for Rating:\n",
      " - High Rating (4-5): Reserved for responses that are very close to the quality of the reference or even better.\n",
      " - Medium Rating (3): Reserved for responses that have moderate quality compared to the reference.\n",
      " - Low Rating (1-2): Allocated to response that are much lower quality compared to the reference or completely wrong.\n",
      "\n",
      "Assessment:\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from src.utils import inspect_llm_judge_queries\n",
    "\n",
    "inspect_llm_judge_queries(dataset_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we apply a similar online batch-inference method to generate our labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting batch inference on 30 queries...\n",
      "# queries un-processed: 29, in-progress: 1, ready: 0\n",
      "# queries un-processed: 28, in-progress: 2, ready: 0\n",
      "# queries un-processed: 27, in-progress: 3, ready: 0\n",
      "# queries un-processed: 26, in-progress: 4, ready: 0\n",
      "# queries un-processed: 25, in-progress: 5, ready: 0\n",
      "# queries un-processed: 24, in-progress: 6, ready: 0\n",
      "# queries un-processed: 23, in-progress: 6, ready: 1\n",
      "# queries un-processed: 22, in-progress: 6, ready: 1\n",
      "# queries un-processed: 21, in-progress: 7, ready: 0\n",
      "# queries un-processed: 20, in-progress: 7, ready: 1\n",
      "# queries un-processed: 19, in-progress: 7, ready: 1\n",
      "# queries un-processed: 18, in-progress: 8, ready: 0\n",
      "# queries un-processed: 17, in-progress: 9, ready: 0\n",
      "# queries un-processed: 16, in-progress: 9, ready: 1\n",
      "# queries un-processed: 15, in-progress: 9, ready: 1\n",
      "# queries un-processed: 14, in-progress: 9, ready: 1\n",
      "# queries un-processed: 13, in-progress: 10, ready: 0\n",
      "# queries un-processed: 13, in-progress: 10, ready: 0\n",
      "# queries un-processed: 13, in-progress: 9, ready: 1\n",
      "# queries un-processed: 12, in-progress: 9, ready: 1\n",
      "# queries un-processed: 11, in-progress: 9, ready: 1\n",
      "# queries un-processed: 10, in-progress: 9, ready: 1\n",
      "# queries un-processed: 9, in-progress: 9, ready: 1\n",
      "# queries un-processed: 8, in-progress: 9, ready: 1\n",
      "# queries un-processed: 7, in-progress: 9, ready: 1\n",
      "# queries un-processed: 6, in-progress: 9, ready: 1\n",
      "# queries un-processed: 5, in-progress: 9, ready: 1\n",
      "# queries un-processed: 4, in-progress: 9, ready: 1\n",
      "# queries un-processed: 3, in-progress: 10, ready: 0\n",
      "# queries un-processed: 3, in-progress: 9, ready: 1\n",
      "# queries un-processed: 2, in-progress: 10, ready: 0\n",
      "# queries un-processed: 2, in-progress: 10, ready: 0\n",
      "# queries un-processed: 2, in-progress: 9, ready: 1\n",
      "# queries un-processed: 1, in-progress: 9, ready: 1\n",
      "# queries un-processed: 0, in-progress: 9, ready: 1\n",
      "# queries un-processed: 0, in-progress: 8, ready: 1\n",
      "# queries un-processed: 0, in-progress: 7, ready: 1\n",
      "# queries un-processed: 0, in-progress: 6, ready: 1\n",
      "# queries un-processed: 0, in-progress: 6, ready: 0\n",
      "# queries un-processed: 0, in-progress: 6, ready: 0\n",
      "# queries un-processed: 0, in-progress: 5, ready: 1\n",
      "# queries un-processed: 0, in-progress: 4, ready: 1\n",
      "# queries un-processed: 0, in-progress: 3, ready: 1\n",
      "# queries un-processed: 0, in-progress: 2, ready: 1\n",
      "# queries un-processed: 0, in-progress: 2, ready: 0\n",
      "# queries un-processed: 0, in-progress: 2, ready: 0\n",
      "# queries un-processed: 0, in-progress: 1, ready: 1\n",
      "# queries un-processed: 0, in-progress: 1, ready: 0\n",
      "# queries un-processed: 0, in-progress: 1, ready: 0\n",
      "# queries un-processed: 0, in-progress: 0, ready: 1\n",
      "Done in 15.44sec.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from src.online_inference import generate_llm_judge_labels\n",
    "\n",
    "dataset_df = generate_llm_judge_labels(dataset_df, os.getenv('OPENAI_API_KEY'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset overview with score labels\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>source</th>\n",
       "      <th>gpt4_response</th>\n",
       "      <th>mixtral_response</th>\n",
       "      <th>mixtral_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6062</th>\n",
       "      <td>Based on the features mentioned, which hotel d...</td>\n",
       "      <td>[evol_instruct]</td>\n",
       "      <td>Based on the features mentioned, Hotel A seems...</td>\n",
       "      <td>Based on the information provided, I would sa...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>113830</th>\n",
       "      <td>Provide step-by-step instructions on how to cr...</td>\n",
       "      <td>[ultrachat]</td>\n",
       "      <td>Sure, here's a simple step-by-step guide on ho...</td>\n",
       "      <td>Sure, I'd be happy to help you make a homemad...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138869</th>\n",
       "      <td>What are the 10 largest cities in the US by po...</td>\n",
       "      <td>[lmsys-chat-1m]</td>\n",
       "      <td>As of the most recent data available, the 10 l...</td>\n",
       "      <td>Sure, I'd be happy to help with that! Here ar...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169249</th>\n",
       "      <td>Write a comparison essay of at least 500 words...</td>\n",
       "      <td>[ultrachat]</td>\n",
       "      <td>Title: A Comparative Analysis of Driving a Car...</td>\n",
       "      <td>Title: The Great Debate: Driving a Car vs. Ri...</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116934</th>\n",
       "      <td>Q: You are provided with an \"Event\", \"Intent\" ...</td>\n",
       "      <td>[flan_v2_niv2]</td>\n",
       "      <td>PersonX might feel satisfied or content using ...</td>\n",
       "      <td>PersonX probably feels comfortable and focuse...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   prompt           source  \\\n",
       "6062    Based on the features mentioned, which hotel d...  [evol_instruct]   \n",
       "113830  Provide step-by-step instructions on how to cr...      [ultrachat]   \n",
       "138869  What are the 10 largest cities in the US by po...  [lmsys-chat-1m]   \n",
       "169249  Write a comparison essay of at least 500 words...      [ultrachat]   \n",
       "116934  Q: You are provided with an \"Event\", \"Intent\" ...   [flan_v2_niv2]   \n",
       "\n",
       "                                            gpt4_response  \\\n",
       "6062    Based on the features mentioned, Hotel A seems...   \n",
       "113830  Sure, here's a simple step-by-step guide on ho...   \n",
       "138869  As of the most recent data available, the 10 l...   \n",
       "169249  Title: A Comparative Analysis of Driving a Car...   \n",
       "116934  PersonX might feel satisfied or content using ...   \n",
       "\n",
       "                                         mixtral_response  mixtral_score  \n",
       "6062     Based on the information provided, I would sa...              5  \n",
       "113830   Sure, I'd be happy to help you make a homemad...              3  \n",
       "138869   Sure, I'd be happy to help with that! Here ar...              5  \n",
       "169249   Title: The Great Debate: Driving a Car vs. Ri...              4  \n",
       "116934   PersonX probably feels comfortable and focuse...              5  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(dataset_df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Full Dataset\n",
    "We have previously generated the full labeled datasets, created a train and validation splits, and published them as a public huggingface dataset `routellm/gpt4_dataset`. Let's load the dataset and explore the score distribution.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train size: 109101\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>source</th>\n",
       "      <th>gpt4_response</th>\n",
       "      <th>mixtral_response</th>\n",
       "      <th>mixtral_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>I'll give you a review, can you extract the fo...</td>\n",
       "      <td>[lmsys-chat-1m]</td>\n",
       "      <td>Sure, here's the analysis of the review:\\n\\n1....</td>\n",
       "      <td>Food aspects and opinion words:\\n\\n1. Made to ...</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Answer the following question: Claim: \"Joker m...</td>\n",
       "      <td>[flan_v2_cot]</td>\n",
       "      <td>The answer is no.\\nChain of thoughts: Stan Lee...</td>\n",
       "      <td>The answer is no.\\n\\nChain of thoughts: While ...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>TASK DEFINITION: In this task you will be give...</td>\n",
       "      <td>[flan_v2_niv2]</td>\n",
       "      <td>ZdoublexpropheciesS</td>\n",
       "      <td>ZdoublexpropheciesS\\n\\nIn this task, you are a...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Detailed Instructions: In this task, you need ...</td>\n",
       "      <td>[flan_v2_niv2]</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No, 'station' is not the longest word in the s...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A guy pick up a woman Then he puts her down Ex...</td>\n",
       "      <td>[sharegpt]</td>\n",
       "      <td>This phrase could be interpreted as a joke bec...</td>\n",
       "      <td>This joke is a play on words and relies on the...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt           source  \\\n",
       "0  I'll give you a review, can you extract the fo...  [lmsys-chat-1m]   \n",
       "1  Answer the following question: Claim: \"Joker m...    [flan_v2_cot]   \n",
       "2  TASK DEFINITION: In this task you will be give...   [flan_v2_niv2]   \n",
       "3  Detailed Instructions: In this task, you need ...   [flan_v2_niv2]   \n",
       "4  A guy pick up a woman Then he puts her down Ex...       [sharegpt]   \n",
       "\n",
       "                                       gpt4_response  \\\n",
       "0  Sure, here's the analysis of the review:\\n\\n1....   \n",
       "1  The answer is no.\\nChain of thoughts: Stan Lee...   \n",
       "2                                ZdoublexpropheciesS   \n",
       "3                                                Yes   \n",
       "4  This phrase could be interpreted as a joke bec...   \n",
       "\n",
       "                                    mixtral_response  mixtral_score  \n",
       "0  Food aspects and opinion words:\\n\\n1. Made to ...              4  \n",
       "1  The answer is no.\\n\\nChain of thoughts: While ...              5  \n",
       "2  ZdoublexpropheciesS\\n\\nIn this task, you are a...              5  \n",
       "3  No, 'station' is not the longest word in the s...              5  \n",
       "4  This joke is a play on words and relies on the...              5  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk0AAAHHCAYAAACiOWx7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAABGVklEQVR4nO3dfXgNd/7/8ddJyI1wTtwmskIoRVo3FURKtdpUaPTLlpau1birpaEidbtVd1/K6g1ad9vtrtjd2tL9lm6lQkTRrbiLalHJasvSkqAkhywJyfz+6C+zjgQTDSfh+biuuep8Pu8z8545euVlzszEZhiGIQAAAFyXh7sbAAAAqAgITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE1AORcSEqKBAwe6u4073muvvaZGjRrJ09NTrVu3vmXbSUhIkM1m05EjR27ZNm5Gee0LKE8ITcBtVPSDaffu3SXOP/LII7r//vt/9nY++eQTTZs27Wev526xYcMGjR8/Xh07dtSyZcv06quvurulEr366qtas2aNu9sA7lqV3N0AgOvLyMiQh0fp/n3zySefaNGiRQQnizZt2iQPDw/98Y9/lJeX1y3d1oABA9SvXz95e3uX+r2vvvqq+vTpo169epV9YwBuiDNNQDnn7e2typUru7uNUsnNzXV3C6Vy8uRJ+fr63vLAJEmenp7y8fGRzWa7pdupaJ/Bzbpb9hPlA6EJKOeuvqbp0qVLmj59upo0aSIfHx/VrFlTnTp1UnJysiRp4MCBWrRokSTJZrOZS5Hc3Fy99NJLCg4Olre3t5o2barXX39dhmG4bPfChQt68cUXVatWLVWrVk3/8z//ox9++EE2m83lDNa0adNks9n09ddf61e/+pWqV6+uTp06SZK++uorDRw4UI0aNZKPj48CAwM1ePBg/fjjjy7bKlrHv/71L/3617+Ww+FQ7dq19corr8gwDB07dkw9e/aU3W5XYGCg3njjDUvH7vLly/rf//1f3XPPPfL29lZISIh++9vfKi8vz6yx2WxatmyZcnNzzWOVkJBwzXUWfYX61Vdf6eGHH1aVKlXUuHFj/f3vf5ckbdmyReHh4fL19VXTpk21ceNGl/dffe1Q0VmuKVOmuNStWLFCNptNS5YsMfvMzc3V8uXLzT6L/l6UxWdwM86dO6e4uDiFhITI29tbderU0eOPP649e/a41O3YsUNPPPGEqlevLj8/P7Vs2VILFixwqdm0aZMeeugh+fn5yd/fXz179tTBgwddaq63n5L017/+VWFhYfL19VWNGjXUr18/HTt27GfvJ1CEr+cAN8jJydHp06eLjV+6dOmG7502bZpmz56toUOHqn379nI6ndq9e7f27Nmjxx9/XL/5zW90/PhxJScn6y9/+YvLew3D0P/8z//o008/1ZAhQ9S6dWutX79e48aN0w8//KB58+aZtQMHDtSqVas0YMAAdejQQVu2bFF0dPQ1+3r66afVpEkTvfrqq2YAS05O1nfffadBgwYpMDBQBw4c0DvvvKMDBw5o+/btxc629O3bV82bN9ecOXOUmJiomTNnqkaNGvr973+vRx99VL/73e/03nvvaezYsWrXrp06d+583WM1dOhQLV++XH369NFLL72kHTt2aPbs2Tp48KBWr14tSfrLX/6id955Rzt37tS7774rSXrwwQevu96zZ8+qR48e6tevn55++mktWbJE/fr103vvvae4uDgNHz5cv/rVr/Taa6+pT58+OnbsmKpVq1biuh599FG98MILmj17tnr16qU2bdroxIkTGjVqlCIjIzV8+HCzz6LPfNiwYZKke+65p8w/g9IYPny4/v73v2vkyJEKDQ3Vjz/+qH/+8586ePCg2rRpY26/R48eqlu3rkaPHq3AwEAdPHhQa9eu1ejRoyVJGzduVPfu3dWoUSNNmzZNFy5c0Ntvv62OHTtqz549CgkJueF+zpo1S6+88oqeeeYZDR06VKdOndLbb7+tzp0764svvpC/v/9N7ydgMgDcNsuWLTMkXXe57777XN7ToEEDIyYmxnzdqlUrIzo6+rrbiY2NNUr633vNmjWGJGPmzJku43369DFsNpvxzTffGIZhGGlpaYYkIy4uzqVu4MCBhiRj6tSp5tjUqVMNScazzz5bbHv/+c9/io397W9/MyQZW7duLbaOYcOGmWOXL1826tWrZ9hsNmPOnDnm+NmzZw1fX1+XY1KSvXv3GpKMoUOHuoyPHTvWkGRs2rTJHIuJiTH8/Pyuu74iDz/8sCHJWLFihTmWnp5uSDI8PDyM7du3m+Pr1683JBnLli0zx4r+Dhw+fNgcy83NNRo3bmzcd999xsWLF43o6GjDbrcb//73v1227efnV+J+l8VnUFJfN+JwOIzY2Nhrzl++fNlo2LCh0aBBA+Ps2bMuc4WFheafW7dubdSpU8f48ccfzbEvv/zS8PDwMJ577jlz7Fr7eeTIEcPT09OYNWuWy/i+ffuMSpUqFRsHbhZfzwFusGjRIiUnJxdbWrZsecP3+vv768CBAzp06FCpt/vJJ5/I09NTL774osv4Sy+9JMMwtG7dOklSUlKSJOmFF15wqRs1atQ11110RuRKvr6+5p8vXryo06dPq0OHDpJU7Csc6aczQ0U8PT3Vtm1bGYahIUOGmOP+/v5q2rSpvvvuu2v2Iv20r5IUHx/vMv7SSy9JkhITE6/7/uupWrWq+vXrZ75u2rSp/P391bx5c4WHh5vjRX++Ua9VqlRRQkKCDh48qM6dOysxMVHz5s1T/fr1S9VXWXwGpeHv768dO3bo+PHjJc5/8cUXOnz4sOLi4oqd6Sk6w3XixAnt3btXAwcOVI0aNcz5li1b6vHHHzc/xytdvZ8ffvihCgsL9cwzz+j06dPmEhgYqCZNmujTTz/9WfsJFOHrOcAN2rdvr7Zt2xYbr169eolf211pxowZ6tmzp+69917df//96tatmwYMGGApcP373/9WUFBQsa+Kmjdvbs4X/dfDw0MNGzZ0qWvcuPE11311rSSdOXNG06dP1/vvv6+TJ0+6zOXk5BSrvzokOBwO+fj4qFatWsXGb3RNTtE+XN1zYGCg/P39zX29GfXq1Sv2tZbD4VBwcHCxMemnr/NupGPHjhoxYoQWLVqkqKgoDR48uNR9lcVnUBpz585VTEyMgoODFRYWpieeeELPPfecGjVqJEn69ttvJem6j9Eo+hyaNm1abK558+Zav369cnNz5efnZ45fvZ+HDh2SYRhq0qRJiduoaDdSoPwiNAEVTOfOnfXtt9/qo48+0oYNG/Tuu+9q3rx5Wrp0qcuZmtvtyjMaRZ555hlt27ZN48aNU+vWrVW1alUVFhaqW7duKiwsLFbv6elpaUxSsQvXr+VW3KV2rZ5+Tq95eXnavHmzpJ/Cxn/+8x9VqVKlVH2VxWdQGs8884weeughrV69Whs2bNBrr72m3/3ud/rwww/VvXv3n7Xu67l6PwsLC2Wz2bRu3boSP4OqVavesl5wdyE0ARVQjRo1NGjQIA0aNEjnz59X586dNW3aNDM0XSsoNGjQQBs3btS5c+dczjalp6eb80X/LSws1OHDh13+9f7NN99Y7vHs2bNKSUnR9OnTXe4Mu5mvFW9G0T4cOnTIPJMmSVlZWcrOzjb3tbyYOnWqDh48qNdff10TJkzQxIkT9dZbb7nUlDYA3o7PoG7dunrhhRf0wgsv6OTJk2rTpo1mzZql7t27mxeq79+/X5GRkSW+v+hzyMjIKDaXnp6uWrVquZxlKsk999wjwzDUsGFD3XvvvT9zj4Br45omoIK5+mupqlWrqnHjxi630Rf9kMnOznapfeKJJ1RQUKCFCxe6jM+bN082m808OxAVFSVJWrx4sUvd22+/bbnPon/xX32WZf78+ZbX8XM88cQTJW7vzTfflKTr3gl4u+3YsUOvv/664uLi9NJLL2ncuHFauHChtmzZ4lLn5+dX7DO9nlv5GRQUFBT7eq9OnToKCgoy/y62adNGDRs21Pz584v1XdRT3bp11bp1ay1fvtylZv/+/dqwYYP5OV7PU089JU9PT02fPr3YvhqGUSaPVwAkzjQBFU5oaKgeeeQRhYWFqUaNGtq9e7d523eRsLAwSdKLL76oqKgoeXp6ql+/fnryySfVpUsXvfzyyzpy5IhatWqlDRs26KOPPlJcXJx5ZiAsLEy9e/fW/Pnz9eOPP5qPHPjXv/4lydoZD7vdrs6dO2vu3Lm6dOmSfvGLX2jDhg06fPjwLTgqxbVq1UoxMTF65513lJ2drYcfflg7d+7U8uXL1atXL3Xp0uW29HEjFy9eVExMjJo0aaJZs2ZJkqZPn66PP/5YgwYN0r59+8wQHBYWpo0bN+rNN99UUFCQGjZs6HLh+dVu5Wdw7tw51atXT3369FGrVq1UtWpVbdy4Ubt27TKfo+Xh4aElS5boySefVOvWrTVo0CDVrVtX6enpOnDggNavXy/pp9/71717d0VERGjIkCHmIwccDoelp9rfc889mjlzpiZNmqQjR46oV69eqlatmg4fPqzVq1dr2LBhGjt27M/eZ4BHDgC3UdFt3bt27Spx/uGHH77hIwdmzpxptG/f3vD39zd8fX2NZs2aGbNmzTLy8/PNmsuXLxujRo0yateubdhsNpfHD5w7d84YM2aMERQUZFSuXNlo0qSJ8dprr7ncAm4YP90GHxsba9SoUcOoWrWq0atXLyMjI8OQ5PIIgKLbwE+dOlVsf77//nvjl7/8peHv7284HA7j6aefNo4fP37NxxZcvY5rPQqgpONUkkuXLhnTp083GjZsaFSuXNkIDg42Jk2aZFy8eNHSdkpyrW03aNCgxEdBSHK5Lf/qW/vHjBljeHp6Gjt27HB53+7du41KlSoZI0aMMMfS09ONzp07G76+voYk8+9FWXwGpX3kQF5enjFu3DijVatWRrVq1Qw/Pz+jVatWxuLFi4vV/vOf/zQef/xxs65ly5bG22+/7VKzceNGo2PHjoavr69ht9uNJ5980vj6669daq63n4ZhGP/3f/9ndOrUyfDz8zP8/PyMZs2aGbGxsUZGRoalfQJuxGYYFq+mBHDX27t3rx544AH99a9/Vf/+/d3dDgDcVlzTBKBEFy5cKDY2f/58eXh43PBJ3ABwJ+KaJgAlmjt3rtLS0tSlSxdVqlRJ69at07p16zRs2LBizyPCneX8+fM6f/78dWtq1659zUcsAHcqvp4DUKLk5GRNnz5dX3/9tc6fP6/69etrwIABevnll1WpEv/eupNNmzZN06dPv27N4cOHi/1OOOBOR2gCALj47rvvbvirXzp16iQfH5/b1BFQPhCaAAAALOBCcAAAAAu4MKGMFBYW6vjx46pWrdot+V1XAACg7BmGoXPnzikoKEgeHtc/l0RoKiPHjx/njiIAACqoY8eOqV69etetITSVkaJffnrs2DHZ7XY3dwMAAKxwOp0KDg52+SXm10JoKiNFX8nZ7XZCEwAAFYyVS2u4EBwAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsKCSuxsAAOBuFTIx0d0tVChH5kS7dfucaQIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsMCtoSkkJEQ2m63YEhsbK0m6ePGiYmNjVbNmTVWtWlW9e/dWVlaWyzqOHj2q6OhoValSRXXq1NG4ceN0+fJll5rNmzerTZs28vb2VuPGjZWQkFCsl0WLFikkJEQ+Pj4KDw/Xzp07b9l+AwCAisetoWnXrl06ceKEuSQnJ0uSnn76aUnSmDFj9PHHH+uDDz7Qli1bdPz4cT311FPm+wsKChQdHa38/Hxt27ZNy5cvV0JCgqZMmWLWHD58WNHR0erSpYv27t2ruLg4DR06VOvXrzdrVq5cqfj4eE2dOlV79uxRq1atFBUVpZMnT96mIwEAAMo7m2EYhrubKBIXF6e1a9fq0KFDcjqdql27tlasWKE+ffpIktLT09W8eXOlpqaqQ4cOWrdunXr06KHjx48rICBAkrR06VJNmDBBp06dkpeXlyZMmKDExETt37/f3E6/fv2UnZ2tpKQkSVJ4eLjatWunhQsXSpIKCwsVHBysUaNGaeLEiZZ6dzqdcjgcysnJkd1uL8vDAgC4Q4VMTHR3CxXKkTnRZb7O0vz8LjfXNOXn5+uvf/2rBg8eLJvNprS0NF26dEmRkZFmTbNmzVS/fn2lpqZKklJTU9WiRQszMElSVFSUnE6nDhw4YNZcuY6imqJ15OfnKy0tzaXGw8NDkZGRZk1J8vLy5HQ6XRYAAHDnKjehac2aNcrOztbAgQMlSZmZmfLy8pK/v79LXUBAgDIzM82aKwNT0XzR3PVqnE6nLly4oNOnT6ugoKDEmqJ1lGT27NlyOBzmEhwcXOp9BgAAFUe5CU1//OMf1b17dwUFBbm7FUsmTZqknJwcczl27Ji7WwIAALdQJXc3IEn//ve/tXHjRn344YfmWGBgoPLz85Wdne1ytikrK0uBgYFmzdV3uRXdXXdlzdV33GVlZclut8vX11eenp7y9PQssaZoHSXx9vaWt7d36XcWAABUSOXiTNOyZctUp04dRUf/9wKvsLAwVa5cWSkpKeZYRkaGjh49qoiICElSRESE9u3b53KXW3Jysux2u0JDQ82aK9dRVFO0Di8vL4WFhbnUFBYWKiUlxawBAABw+5mmwsJCLVu2TDExMapU6b/tOBwODRkyRPHx8apRo4bsdrtGjRqliIgIdejQQZLUtWtXhYaGasCAAZo7d64yMzM1efJkxcbGmmeBhg8froULF2r8+PEaPHiwNm3apFWrVikx8b93LMTHxysmJkZt27ZV+/btNX/+fOXm5mrQoEG392AAAIByy+2haePGjTp69KgGDx5cbG7evHny8PBQ7969lZeXp6ioKC1evNic9/T01Nq1azVixAhFRETIz89PMTExmjFjhlnTsGFDJSYmasyYMVqwYIHq1aund999V1FRUWZN3759derUKU2ZMkWZmZlq3bq1kpKSil0cDgAA7l7l6jlNFRnPaQIAlBbPaSodntMEAABQARCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALDA7aHphx9+0K9//WvVrFlTvr6+atGihXbv3m3OG4ahKVOmqG7duvL19VVkZKQOHTrkso4zZ86of//+stvt8vf315AhQ3T+/HmXmq+++koPPfSQfHx8FBwcrLlz5xbr5YMPPlCzZs3k4+OjFi1a6JNPPrk1Ow0AACoct4ams2fPqmPHjqpcubLWrVunr7/+Wm+88YaqV69u1sydO1dvvfWWli5dqh07dsjPz09RUVG6ePGiWdO/f38dOHBAycnJWrt2rbZu3aphw4aZ806nU127dlWDBg2Ulpam1157TdOmTdM777xj1mzbtk3PPvushgwZoi+++EK9evVSr169tH///ttzMAAAQLlmMwzDcNfGJ06cqM8//1yfffZZifOGYSgoKEgvvfSSxo4dK0nKyclRQECAEhIS1K9fPx08eFChoaHatWuX2rZtK0lKSkrSE088oe+//15BQUFasmSJXn75ZWVmZsrLy8vc9po1a5Seni5J6tu3r3Jzc7V27Vpz+x06dFDr1q21dOnSG+6L0+mUw+FQTk6O7Hb7zzouAIC7Q8jERHe3UKEcmRNd5usszc9vt55p+sc//qG2bdvq6aefVp06dfTAAw/oD3/4gzl/+PBhZWZmKjIy0hxzOBwKDw9XamqqJCk1NVX+/v5mYJKkyMhIeXh4aMeOHWZN586dzcAkSVFRUcrIyNDZs2fNmiu3U1RTtJ2r5eXlyel0uiwAAODO5dbQ9N1332nJkiVq0qSJ1q9frxEjRujFF1/U8uXLJUmZmZmSpICAAJf3BQQEmHOZmZmqU6eOy3ylSpVUo0YNl5qS1nHlNq5VUzR/tdmzZ8vhcJhLcHBwqfcfAABUHG4NTYWFhWrTpo1effVVPfDAAxo2bJief/55S1+HudukSZOUk5NjLseOHXN3SwAA4BZya2iqW7euQkNDXcaaN2+uo0ePSpICAwMlSVlZWS41WVlZ5lxgYKBOnjzpMn/58mWdOXPGpaakdVy5jWvVFM1fzdvbW3a73WUBAAB3LreGpo4dOyojI8Nl7F//+pcaNGggSWrYsKECAwOVkpJizjudTu3YsUMRERGSpIiICGVnZystLc2s2bRpkwoLCxUeHm7WbN26VZcuXTJrkpOT1bRpU/NOvYiICJftFNUUbQcAANzd3BqaxowZo+3bt+vVV1/VN998oxUrVuidd95RbGysJMlmsykuLk4zZ87UP/7xD+3bt0/PPfecgoKC1KtXL0k/nZnq1q2bnn/+ee3cuVOff/65Ro4cqX79+ikoKEiS9Ktf/UpeXl4aMmSIDhw4oJUrV2rBggWKj483exk9erSSkpL0xhtvKD09XdOmTdPu3bs1cuTI235cAABA+VPJnRtv166dVq9erUmTJmnGjBlq2LCh5s+fr/79+5s148ePV25uroYNG6bs7Gx16tRJSUlJ8vHxMWvee+89jRw5Uo899pg8PDzUu3dvvfXWW+a8w+HQhg0bFBsbq7CwMNWqVUtTpkxxeZbTgw8+qBUrVmjy5Mn67W9/qyZNmmjNmjW6//77b8/BAAAA5Zpbn9N0J+E5TQCA0uI5TaVzVz+nCQAAoKIgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABggVtD07Rp02Sz2VyWZs2amfMXL15UbGysatasqapVq6p3797KyspyWcfRo0cVHR2tKlWqqE6dOho3bpwuX77sUrN582a1adNG3t7eaty4sRISEor1smjRIoWEhMjHx0fh4eHauXPnLdlnAABQMbn9TNN9992nEydOmMs///lPc27MmDH6+OOP9cEHH2jLli06fvy4nnrqKXO+oKBA0dHRys/P17Zt27R8+XIlJCRoypQpZs3hw4cVHR2tLl26aO/evYqLi9PQoUO1fv16s2blypWKj4/X1KlTtWfPHrVq1UpRUVE6efLk7TkIAACg3LMZhmG4a+PTpk3TmjVrtHfv3mJzOTk5ql27tlasWKE+ffpIktLT09W8eXOlpqaqQ4cOWrdunXr06KHjx48rICBAkrR06VJNmDBBp06dkpeXlyZMmKDExETt37/fXHe/fv2UnZ2tpKQkSVJ4eLjatWunhQsXSpIKCwsVHBysUaNGaeLEiZb2xel0yuFwKCcnR3a7/eccFgDAXSJkYqK7W6hQjsyJLvN1lubnt9vPNB06dEhBQUFq1KiR+vfvr6NHj0qS0tLSdOnSJUVGRpq1zZo1U/369ZWamipJSk1NVYsWLczAJElRUVFyOp06cOCAWXPlOopqitaRn5+vtLQ0lxoPDw9FRkaaNSXJy8uT0+l0WQAAwJ3LraEpPDxcCQkJSkpK0pIlS3T48GE99NBDOnfunDIzM+Xl5SV/f3+X9wQEBCgzM1OSlJmZ6RKYiuaL5q5X43Q6deHCBZ0+fVoFBQUl1hStoySzZ8+Ww+Ewl+Dg4Js6BgAAoGKo5M6Nd+/e3fxzy5YtFR4ergYNGmjVqlXy9fV1Y2c3NmnSJMXHx5uvnU4nwQkAgDuY27+eu5K/v7/uvfdeffPNNwoMDFR+fr6ys7NdarKyshQYGChJCgwMLHY3XdHrG9XY7Xb5+vqqVq1a8vT0LLGmaB0l8fb2lt1ud1kAAMCdq1yFpvPnz+vbb79V3bp1FRYWpsqVKyslJcWcz8jI0NGjRxURESFJioiI0L59+1zucktOTpbdbldoaKhZc+U6imqK1uHl5aWwsDCXmsLCQqWkpJg1AAAAbg1NY8eO1ZYtW3TkyBFt27ZNv/zlL+Xp6alnn31WDodDQ4YMUXx8vD799FOlpaVp0KBBioiIUIcOHSRJXbt2VWhoqAYMGKAvv/xS69ev1+TJkxUbGytvb29J0vDhw/Xdd99p/PjxSk9P1+LFi7Vq1SqNGTPG7CM+Pl5/+MMftHz5ch08eFAjRoxQbm6uBg0a5JbjAgAAyh+3XtP0/fff69lnn9WPP/6o2rVrq1OnTtq+fbtq164tSZo3b548PDzUu3dv5eXlKSoqSosXLzbf7+npqbVr12rEiBGKiIiQn5+fYmJiNGPGDLOmYcOGSkxM1JgxY7RgwQLVq1dP7777rqKiosyavn376tSpU5oyZYoyMzPVunVrJSUlFbs4HAAA3L3c+pymOwnPaQIAlBbPaSqdu/45TQAAABUBoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMCCmwpNjRo10o8//lhsPDs7W40aNfrZTQEAAJQ3NxWajhw5ooKCgmLjeXl5+uGHH352UwAAAOVNpdIU/+Mf/zD/vH79ejkcDvN1QUGBUlJSFBISUmbNAQAAlBelCk29evWSJNlsNsXExLjMVa5cWSEhIXrjjTfKrDkAAIDyolShqbCwUJLUsGFD7dq1S7Vq1bolTQEAAJQ3pQpNRQ4fPlzWfQAAAJRrNxWaJCklJUUpKSk6efKkeQaqyJ/+9Kef3RgAAEB5clOhafr06ZoxY4batm2runXrymazlXVfAAAA5cpNhaalS5cqISFBAwYMKOt+AAAAyqWbek5Tfn6+HnzwwbLuBQAAoNy6qdA0dOhQrVixokwbmTNnjmw2m+Li4syxixcvKjY2VjVr1lTVqlXVu3dvZWVlubzv6NGjio6OVpUqVVSnTh2NGzdOly9fdqnZvHmz2rRpI29vbzVu3FgJCQnFtr9o0SKFhITIx8dH4eHh2rlzZ5nuHwAAqNhu6uu5ixcv6p133tHGjRvVsmVLVa5c2WX+zTffLNX6du3apd///vdq2bKly/iYMWOUmJioDz74QA6HQyNHjtRTTz2lzz//XNJPD9SMjo5WYGCgtm3bphMnTui5555T5cqV9eqrr0r66U6/6OhoDR8+XO+9955SUlI0dOhQ1a1bV1FRUZKklStXKj4+XkuXLlV4eLjmz5+vqKgoZWRkqE6dOjdziAAAwB3GZhiGUdo3denS5dortNm0adMmy+s6f/682rRpo8WLF2vmzJlq3bq15s+fr5ycHNWuXVsrVqxQnz59JEnp6elq3ry5UlNT1aFDB61bt049evTQ8ePHFRAQIOmn660mTJigU6dOycvLSxMmTFBiYqL2799vbrNfv37Kzs5WUlKSJCk8PFzt2rXTwoULJf30PKrg4GCNGjVKEydOtLQfTqdTDodDOTk5stvtlvcfAHD3CpmY6O4WKpQjc6LLfJ2l+fl9U2eaPv3005tqrCSxsbGKjo5WZGSkZs6caY6npaXp0qVLioyMNMeaNWum+vXrm6EpNTVVLVq0MAOTJEVFRWnEiBE6cOCAHnjgAaWmprqso6im6GvA/Px8paWladKkSea8h4eHIiMjlZqaes2+8/LylJeXZ752Op03fQwAAED5d9PPaSoL77//vvbs2aNdu3YVm8vMzJSXl5f8/f1dxgMCApSZmWnWXBmYiuaL5q5X43Q6deHCBZ09e1YFBQUl1qSnp1+z99mzZ2v69OnWdhQAAFR4NxWaunTpct1nM1n5eu7YsWMaPXq0kpOT5ePjczNtuNWkSZMUHx9vvnY6nQoODnZjRwAA4Fa6qdDUunVrl9eXLl3S3r17tX///mK/yPda0tLSdPLkSbVp08YcKygo0NatW7Vw4UKtX79e+fn5ys7OdjnblJWVpcDAQElSYGBgsbvciu6uu7Lm6jvusrKyZLfb5evrK09PT3l6epZYU7SOknh7e8vb29vSvgIAgIrvpkLTvHnzShyfNm2azp8/b2kdjz32mPbt2+cyNmjQIDVr1kwTJkxQcHCwKleurJSUFPXu3VuSlJGRoaNHjyoiIkKSFBERoVmzZunkyZPmXW7Jycmy2+0KDQ01az755BOX7SQnJ5vr8PLyUlhYmFJSUtSrVy9JP10InpKSopEjR1raFwAAcOcr02uafv3rX6t9+/Z6/fXXb1hbrVo13X///S5jfn5+qlmzpjk+ZMgQxcfHq0aNGrLb7Ro1apQiIiLUoUMHSVLXrl0VGhqqAQMGaO7cucrMzNTkyZMVGxtrngUaPny4Fi5cqPHjx2vw4MHatGmTVq1apcTE/96xEB8fr5iYGLVt21bt27fX/PnzlZubq0GDBpXVoQEAABVcmYam1NTUMr0+ad68efLw8FDv3r2Vl5enqKgoLV682Jz39PTU2rVrNWLECEVERMjPz08xMTGaMWOGWdOwYUMlJiZqzJgxWrBggerVq6d3333XfEaTJPXt21enTp3SlClTlJmZqdatWyspKanYxeEAAODudVPPaXrqqadcXhuGoRMnTmj37t165ZVXNHXq1DJrsKLgOU0AgNLiOU2lUyGf0+RwOFxee3h4qGnTppoxY4a6du16M6sEAAAo124qNC1btqys+wAAACjXftY1TWlpaTp48KAk6b777tMDDzxQJk0BAACUNzcVmk6ePKl+/fpp8+bN5jOUsrOz1aVLF73//vuqXbt2WfYIAADgdh4386ZRo0bp3LlzOnDggM6cOaMzZ85o//79cjqdevHFF8u6RwAAALe7qTNNSUlJ2rhxo5o3b26OhYaGatGiRVwIDgAA7kg3daapsLBQlStXLjZeuXJlFRYW/uymAAAAypubCk2PPvqoRo8erePHj5tjP/zwg8aMGaPHHnuszJoDAAAoL24qNC1cuFBOp1MhISG65557dM8996hhw4ZyOp16++23y7pHAAAAt7upa5qCg4O1Z88ebdy4Uenp6ZKk5s2bKzIyskybAwAAKC9KdaZp06ZNCg0NldPplM1m0+OPP65Ro0Zp1KhRateune677z599tlnt6pXAAAAtylVaJo/f76ef/75En83i8Ph0G9+8xu9+eabZdYcAABAeVGq0PTll1+qW7du15zv2rWr0tLSfnZTAAAA5U2pQlNWVlaJjxooUqlSJZ06depnNwUAAFDelCo0/eIXv9D+/fuvOf/VV1+pbt26P7spAACA8qZUoemJJ57QK6+8oosXLxabu3DhgqZOnaoePXqUWXMAAADlRakeOTB58mR9+OGHuvfeezVy5Eg1bdpUkpSenq5FixapoKBAL7/88i1pFAAAwJ1KFZoCAgK0bds2jRgxQpMmTZJhGJIkm82mqKgoLVq0SAEBAbekUQAAAHcq9cMtGzRooE8++URnz57VN998I8Mw1KRJE1WvXv1W9AcAAFAu3NQTwSWpevXqateuXVn2AgAAUG7d1O+eAwAAuNsQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAAC9wampYsWaKWLVvKbrfLbrcrIiJC69atM+cvXryo2NhY1axZU1WrVlXv3r2VlZXlso6jR48qOjpaVapUUZ06dTRu3DhdvnzZpWbz5s1q06aNvL291bhxYyUkJBTrZdGiRQoJCZGPj4/Cw8O1c+fOW7LPAACgYnJraKpXr57mzJmjtLQ07d69W48++qh69uypAwcOSJLGjBmjjz/+WB988IG2bNmi48eP66mnnjLfX1BQoOjoaOXn52vbtm1avny5EhISNGXKFLPm8OHDio6OVpcuXbR3717FxcVp6NChWr9+vVmzcuVKxcfHa+rUqdqzZ49atWqlqKgonTx58vYdDAAAUK7ZDMMw3N3ElWrUqKHXXntNffr0Ue3atbVixQr16dNHkpSenq7mzZsrNTVVHTp00Lp169SjRw8dP35cAQEBkqSlS5dqwoQJOnXqlLy8vDRhwgQlJiZq//795jb69eun7OxsJSUlSZLCw8PVrl07LVy4UJJUWFio4OBgjRo1ShMnTrTUt9PplMPhUE5Ojux2e1keEgDAHSpkYqK7W6hQjsyJLvN1lubnd7m5pqmgoEDvv/++cnNzFRERobS0NF26dEmRkZFmTbNmzVS/fn2lpqZKklJTU9WiRQszMElSVFSUnE6nebYqNTXVZR1FNUXryM/PV1pamkuNh4eHIiMjzZqS5OXlyel0uiwAAODO5fbQtG/fPlWtWlXe3t4aPny4Vq9erdDQUGVmZsrLy0v+/v4u9QEBAcrMzJQkZWZmugSmovmiuevVOJ1OXbhwQadPn1ZBQUGJNUXrKMns2bPlcDjMJTg4+Kb2HwAAVAxuD01NmzbV3r17tWPHDo0YMUIxMTH6+uuv3d3WDU2aNEk5OTnmcuzYMXe3BAAAbqFK7m7Ay8tLjRs3liSFhYVp165dWrBggfr27av8/HxlZ2e7nG3KyspSYGCgJCkwMLDYXW5Fd9ddWXP1HXdZWVmy2+3y9fWVp6enPD09S6wpWkdJvL295e3tfXM7DQAAKhy3n2m6WmFhofLy8hQWFqbKlSsrJSXFnMvIyNDRo0cVEREhSYqIiNC+fftc7nJLTk6W3W5XaGioWXPlOopqitbh5eWlsLAwl5rCwkKlpKSYNQAAAG490zRp0iR1795d9evX17lz57RixQpt3rxZ69evl8Ph0JAhQxQfH68aNWrIbrdr1KhRioiIUIcOHSRJXbt2VWhoqAYMGKC5c+cqMzNTkydPVmxsrHkWaPjw4Vq4cKHGjx+vwYMHa9OmTVq1apUSE/97x0J8fLxiYmLUtm1btW/fXvPnz1dubq4GDRrkluMCAADKH7eGppMnT+q5557TiRMn5HA41LJlS61fv16PP/64JGnevHny8PBQ7969lZeXp6ioKC1evNh8v6enp9auXasRI0YoIiJCfn5+iomJ0YwZM8yahg0bKjExUWPGjNGCBQtUr149vfvuu4qKijJr+vbtq1OnTmnKlCnKzMxU69atlZSUVOzicAAAcPcqd89pqqh4ThMAoLR4TlPp8JwmAACACoDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFbg1Ns2fPVrt27VStWjXVqVNHvXr1UkZGhkvNxYsXFRsbq5o1a6pq1arq3bu3srKyXGqOHj2q6OhoValSRXXq1NG4ceN0+fJll5rNmzerTZs28vb2VuPGjZWQkFCsn0WLFikkJEQ+Pj4KDw/Xzp07y3yfAQBAxeTW0LRlyxbFxsZq+/btSk5O1qVLl9S1a1fl5uaaNWPGjNHHH3+sDz74QFu2bNHx48f11FNPmfMFBQWKjo5Wfn6+tm3bpuXLlyshIUFTpkwxaw4fPqzo6Gh16dJFe/fuVVxcnIYOHar169ebNStXrlR8fLymTp2qPXv2qFWrVoqKitLJkydvz8EAAADlms0wDMPdTRQ5deqU6tSpoy1btqhz587KyclR7dq1tWLFCvXp00eSlJ6erubNmys1NVUdOnTQunXr1KNHDx0/flwBAQGSpKVLl2rChAk6deqUvLy8NGHCBCUmJmr//v3mtvr166fs7GwlJSVJksLDw9WuXTstXLhQklRYWKjg4GCNGjVKEydOvGHvTqdTDodDOTk5stvtZX1oAAB3oJCJie5uoUI5Mie6zNdZmp/f5eqappycHElSjRo1JElpaWm6dOmSIiMjzZpmzZqpfv36Sk1NlSSlpqaqRYsWZmCSpKioKDmdTh04cMCsuXIdRTVF68jPz1daWppLjYeHhyIjI82aq+Xl5cnpdLosAADgzlVuQlNhYaHi4uLUsWNH3X///ZKkzMxMeXl5yd/f36U2ICBAmZmZZs2VgalovmjuejVOp1MXLlzQ6dOnVVBQUGJN0TquNnv2bDkcDnMJDg6+uR0HAAAVQrkJTbGxsdq/f7/ef/99d7diyaRJk5STk2Mux44dc3dLAADgFqrk7gYkaeTIkVq7dq22bt2qevXqmeOBgYHKz89Xdna2y9mmrKwsBQYGmjVX3+VWdHfdlTVX33GXlZUlu90uX19feXp6ytPTs8SaonVczdvbW97e3je3wwBQDnF9jXW34toalH9uPdNkGIZGjhyp1atXa9OmTWrYsKHLfFhYmCpXrqyUlBRzLCMjQ0ePHlVERIQkKSIiQvv27XO5yy05OVl2u12hoaFmzZXrKKopWoeXl5fCwsJcagoLC5WSkmLWAACAu5tbzzTFxsZqxYoV+uijj1StWjXz+iGHwyFfX185HA4NGTJE8fHxqlGjhux2u0aNGqWIiAh16NBBktS1a1eFhoZqwIABmjt3rjIzMzV58mTFxsaaZ4KGDx+uhQsXavz48Ro8eLA2bdqkVatWKTHxv/+qio+PV0xMjNq2bav27dtr/vz5ys3N1aBBg27/gQEAAOWOW0PTkiVLJEmPPPKIy/iyZcs0cOBASdK8efPk4eGh3r17Ky8vT1FRUVq8eLFZ6+npqbVr12rEiBGKiIiQn5+fYmJiNGPGDLOmYcOGSkxM1JgxY7RgwQLVq1dP7777rqKiosyavn376tSpU5oyZYoyMzPVunVrJSUlFbs4HAAA3J3K1XOaKjKe0wSgouOaJuvK6pomjnnp8JwmAACACoDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFbg1NW7du1ZNPPqmgoCDZbDatWbPGZd4wDE2ZMkV169aVr6+vIiMjdejQIZeaM2fOqH///rLb7fL399eQIUN0/vx5l5qvvvpKDz30kHx8fBQcHKy5c+cW6+WDDz5Qs2bN5OPjoxYtWuiTTz4p8/0FAAAVl1tDU25urlq1aqVFixaVOD937ly99dZbWrp0qXbs2CE/Pz9FRUXp4sWLZk3//v114MABJScna+3atdq6dauGDRtmzjudTnXt2lUNGjRQWlqaXnvtNU2bNk3vvPOOWbNt2zY9++yzGjJkiL744gv16tVLvXr10v79+2/dzgMAgArFZhiG4e4mJMlms2n16tXq1auXpJ/OMgUFBemll17S2LFjJUk5OTkKCAhQQkKC+vXrp4MHDyo0NFS7du1S27ZtJUlJSUl64okn9P333ysoKEhLlizRyy+/rMzMTHl5eUmSJk6cqDVr1ig9PV2S1LdvX+Xm5mrt2rVmPx06dFDr1q21dOlSS/07nU45HA7l5OTIbreX1WEBgNsmZGKiu1uoMI7MiS6T9XDMS6esjvuVSvPzu9xe03T48GFlZmYqMjLSHHM4HAoPD1dqaqokKTU1Vf7+/mZgkqTIyEh5eHhox44dZk3nzp3NwCRJUVFRysjI0NmzZ82aK7dTVFO0nZLk5eXJ6XS6LAAA4M5VbkNTZmamJCkgIMBlPCAgwJzLzMxUnTp1XOYrVaqkGjVquNSUtI4rt3GtmqL5ksyePVsOh8NcgoODS7uLAACgAim3oam8mzRpknJycszl2LFj7m4JAADcQuU2NAUGBkqSsrKyXMazsrLMucDAQJ08edJl/vLlyzpz5oxLTUnruHIb16opmi+Jt7e37Ha7ywIAAO5c5TY0NWzYUIGBgUpJSTHHnE6nduzYoYiICElSRESEsrOzlZaWZtZs2rRJhYWFCg8PN2u2bt2qS5cumTXJyclq2rSpqlevbtZcuZ2imqLtAAAAuDU0nT9/Xnv37tXevXsl/XTx9969e3X06FHZbDbFxcVp5syZ+sc//qF9+/bpueeeU1BQkHmHXfPmzdWtWzc9//zz2rlzpz7//HONHDlS/fr1U1BQkCTpV7/6lby8vDRkyBAdOHBAK1eu1IIFCxQfH2/2MXr0aCUlJemNN95Qenq6pk2bpt27d2vkyJG3+5AAAIByqpI7N75792516dLFfF0UZGJiYpSQkKDx48crNzdXw4YNU3Z2tjp16qSkpCT5+PiY73nvvfc0cuRIPfbYY/Lw8FDv3r311ltvmfMOh0MbNmxQbGyswsLCVKtWLU2ZMsXlWU4PPvigVqxYocmTJ+u3v/2tmjRpojVr1uj++++/DUcBAABUBOXmOU0VHc9pAlDR8cwg63hOk3vwnCYAAIAKgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhQyd0NwJqQiYnubqHCODIn2t0tAADuQIQm4DoIq9YRVgHc6fh6DgAAwALONAEodzjDZx1n+IDbhzNNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE1XWbRokUJCQuTj46Pw8HDt3LnT3S0BAIBygNB0hZUrVyo+Pl5Tp07Vnj171KpVK0VFRenkyZPubg0AALgZoekKb775pp5//nkNGjRIoaGhWrp0qapUqaI//elP7m4NAAC4GaHp/8vPz1daWpoiIyPNMQ8PD0VGRio1NdWNnQEAgPKgkrsbKC9Onz6tgoICBQQEuIwHBAQoPT29WH1eXp7y8vLM1zk5OZIkp9N5S/orzPvPLVnvnagsPwOOu3Ucd/fguLtHWR13jnnp3IqfsUXrNAzjhrWEpps0e/ZsTZ8+vdh4cHCwG7rBlRzz3d3B3Ynj7h4cd/fguLvHrTzu586dk8PhuG4Noen/q1Wrljw9PZWVleUynpWVpcDAwGL1kyZNUnx8vPm6sLBQZ86cUc2aNWWz2W55v+7mdDoVHBysY8eOyW63u7uduwbH3T047u7BcXePu+24G4ahc+fOKSgo6Ia1hKb/z8vLS2FhYUpJSVGvXr0k/RSEUlJSNHLkyGL13t7e8vb2dhnz9/e/DZ2WL3a7/a74n6q84bi7B8fdPTju7nE3HfcbnWEqQmi6Qnx8vGJiYtS2bVu1b99e8+fPV25urgYNGuTu1gAAgJsRmq7Qt29fnTp1SlOmTFFmZqZat26tpKSkYheHAwCAuw+h6SojR44s8es4uPL29tbUqVOLfUWJW4vj7h4cd/fguLsHx/3abIaVe+wAAADucjzcEgAAwAJCEwAAgAWEJgAAAAsITQAAABYQmlAqW7du1ZNPPqmgoCDZbDatWbPG3S3dFWbPnq127dqpWrVqqlOnjnr16qWMjAx3t3XHW7JkiVq2bGk+5C8iIkLr1q1zd1t3lTlz5shmsykuLs7drdzxpk2bJpvN5rI0a9bM3W2VK4QmlEpubq5atWqlRYsWubuVu8qWLVsUGxur7du3Kzk5WZcuXVLXrl2Vm5vr7tbuaPXq1dOcOXOUlpam3bt369FHH1XPnj114MABd7d2V9i1a5d+//vfq2XLlu5u5a5x33336cSJE+byz3/+090tlSs8pwml0r17d3Xv3t3dbdx1kpKSXF4nJCSoTp06SktLU+fOnd3U1Z3vySefdHk9a9YsLVmyRNu3b9d9993npq7uDufPn1f//v31hz/8QTNnznR3O3eNSpUqlfj7VvETzjQBFVBOTo4kqUaNGm7u5O5RUFCg999/X7m5uYqIiHB3O3e82NhYRUdHKzIy0t2t3FUOHTqkoKAgNWrUSP3799fRo0fd3VK5wpkmoIIpLCxUXFycOnbsqPvvv9/d7dzx9u3bp4iICF28eFFVq1bV6tWrFRoa6u627mjvv/++9uzZo127drm7lbtKeHi4EhIS1LRpU504cULTp0/XQw89pP3796tatWrubq9cIDQBFUxsbKz279/PtQa3SdOmTbV3717l5OTo73//u2JiYrRlyxaC0y1y7NgxjR49WsnJyfLx8XF3O3eVKy+9aNmypcLDw9WgQQOtWrVKQ4YMcWNn5QehCahARo4cqbVr12rr1q2qV6+eu9u5K3h5ealx48aSpLCwMO3atUsLFizQ73//ezd3dmdKS0vTyZMn1aZNG3OsoKBAW7du1cKFC5WXlydPT083dnj38Pf317333qtvvvnG3a2UG4QmoAIwDEOjRo3S6tWrtXnzZjVs2NDdLd21CgsLlZeX5+427liPPfaY9u3b5zI2aNAgNWvWTBMmTCAw3Ubnz5/Xt99+qwEDBri7lXKD0IRSOX/+vMu/Og4fPqy9e/eqRo0aql+/vhs7u7PFxsZqxYoV+uijj1StWjVlZmZKkhwOh3x9fd3c3Z1r0qRJ6t69u+rXr69z585pxYoV2rx5s9avX+/u1u5Y1apVK3atnp+fn2rWrMk1fLfY2LFj9eSTT6pBgwY6fvy4pk6dKk9PTz377LPubq3cIDShVHbv3q0uXbqYr+Pj4yVJMTExSkhIcFNXd74lS5ZIkh555BGX8WXLlmngwIG3v6G7xMmTJ/Xcc8/pxIkTcjgcatmypdavX6/HH3/c3a0BZe7777/Xs88+qx9//FG1a9dWp06dtH37dtWuXdvdrZUbNsMwDHc3AQAAUN7xnCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITgHLtkUceUVxc3F27fQDlB6EJwG03cOBA2Ww2DR8+vNhcbGysbDab+aTzDz/8UP/7v/9bqnX36tWrjDoFgP8iNAFwi+DgYL3//vu6cOGCOXbx4kWtWLHC5fcY1qhRQ9WqVSvz7V+6dKnM13k75efnu7sF4K5DaALgFm3atFFwcLA+/PBDc+zDDz9U/fr19cADD5hjV349lp6eripVqmjFihXm/KpVq+Tr66uvv/5a06ZN0/Lly/XRRx/JZrPJZrNp8+bNOnLkiGw2m1auXKmHH35YPj4+eu+99/Tjjz/q2Wef1S9+8QtVqVJFLVq00N/+9reb3qfFixerSZMm8vHxUUBAgPr06WPOFRYWau7cuWrcuLG8vb1Vv359zZo1y5zft2+fHn30Ufn6+qpmzZoaNmyYzp8/b84XnUGbNWuWgoKC1LRpU0nSsWPH9Mwzz8jf3181atRQz549deTIkZveBwDXRmgC4DaDBw/WsmXLzNd/+tOfNGjQoGvWN2vWTK+//rpeeOEFHT16VN9//72GDx+u3/3udwoNDdXYsWP1zDPPqFu3bjpx4oROnDihBx980Hz/xIkTNXr0aB08eFBRUVG6ePGiwsLClJiYqP3792vYsGEaMGCAdu7cWep92b17t1588UXNmDFDGRkZSkpKUufOnc35SZMmac6cOXrllVf09ddfa8WKFQoICJAk5ebmKioqStWrV9euXbv0wQcfaOPGjRo5cqTLNlJSUpSRkaHk5GStXbtWly5dUlRUlKpVq6bPPvtMn3/+uapWrapu3bpxJgq4FQwAuM1iYmKMnj17GidPnjS8vb2NI0eOGEeOHDF8fHyMU6dOGT179jRiYmIMwzCMhx9+2Bg9erTL+6Ojo42HHnrIeOyxx4yuXbsahYWFxdZ9pcOHDxuSjPnz59+wt+joaOOll14yX5e0/ZL83//9n2G32w2n01lszul0Gt7e3sYf/vCHEt/7zjvvGNWrVzfOnz9vjiUmJhoeHh5GZmamuV8BAQFGXl6eWfOXv/zFaNq0qcv+5+XlGb6+vsb69etv2DOA0qnk7tAG4O5Vu3ZtRUdHKyEhQYZhKDo6WrVq1brh+/70pz/p3nvvlYeHhw4cOCCbzWZpe23btnV5XVBQoFdffVWrVq3SDz/8oPz8fOXl5alKlSql3pfHH39cDRo0UKNGjdStWzd169ZNv/zlL1WlShUdPHhQeXl5euyxx0p878GDB9WqVSv5+fmZYx07dlRhYaEyMjLMM1ItWrSQl5eXWfPll1/qm2++KXbN18WLF/Xtt9+Weh8AXB+hCYBbDR482PwaatGiRZbe8+WXXyo3N1ceHh46ceKE6tata+l9V4YSSXrttde0YMECzZ8/Xy1atJCfn5/i4uJu6qutatWqac+ePdq8ebM2bNigKVOmaNq0adq1a5d8fX1LvT4r/Z8/f15hYWF67733itXWrl27TLYJ4L+4pgmAWxVdf1N0fc6NnDlzRgMHDtTLL7+sgQMHqn///i534Hl5eamgoMDStj///HP17NlTv/71r9WqVSs1atRI//rXv256XypVqqTIyEjNnTtXX331lY4cOaJNmzapSZMm8vX1VUpKSonva968uRkEr+zNw8PDvOC7JG3atNGhQ4dUp04dNW7c2GVxOBw3vR8ASkZoAuBWnp6eOnjwoL7++mt5enresH748OEKDg7W5MmT9eabb6qgoEBjx44150NCQvTVV18pIyNDp0+fvu6jBZo0aaLk5GRt27ZNBw8e1G9+8xtlZWXd1H6sXbtWb731lvbu3at///vf+vOf/6zCwkI1bdpUPj4+mjBhgsaPH68///nP+vbbb7V9+3b98Y9/lCT1799fPj4+iomJ0f79+/Xpp59q1KhRGjBggPnVXEn69++vWrVqqWfPnvrss890+PBhbd68WS+++KK+//77m9oPANfG13MA3M5ut1uq+/Of/6xPPvlEX3zxhSpVqqRKlSrpr3/9qzp16qQePXqoe/fuev7557V582a1bdtW58+f16effqqQkJAS1zd58mR99913ioqKUpUqVTRs2DD16tVLOTk5pd4Hf39/ffjhh5o2bZouXryoJk2a6G9/+5vuu+8+SdIrr7yiSpUqacqUKTp+/Ljq1q1rPtyzSpUqWr9+vUaPHq127dqpSpUq6t27t958883rbrNKlSraunWrJkyYoKeeekrnzp3TL37xCz322GOWjykA62yGYRjubgIAAKC84+s5AAAACwhNAGDBZ599pqpVq15zAXDn4+s5ALDgwoUL+uGHH64537hx49vYDQB3IDQBAABYwNdzAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAv+HwJtaiAL9xMDAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "from src.utils import visualize_label_distribution\n",
    "\n",
    "full_dataset_df = load_dataset(\"routellm/gpt4_dataset\")\n",
    "train_df = full_dataset_df[\"train\"].to_pandas()\n",
    "\n",
    "print(f\"Train size: {len(train_df)}\")\n",
    "display(train_df.head())\n",
    "visualize_label_distribution(train_df, key=\"mixtral_score\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Higher counts for 4-5 scores indicate that Mixtral-8x7B consistently produces high-quality responses, demonstrating its competitive performance compared to the June 2023 version of GPT-4, whose responses are logged in the Nectar dataset.\n",
    "\n",
    "Let us assume that if the score is >= 4, we will route to the OSS model (indicating the response quality is good enough); otherwise, we will route to the closed model. Under this assumption, the data distribution looks like this:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk0AAAHHCAYAAACiOWx7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAAA2aElEQVR4nO3deVyVZf7/8fcBZXE5B1eQIsFlTMo0NyTNciIxbSYnKzFz0FzKEUwxTVtcmsrSycwlzbHUKS2r71drXFDC1EbJBcdKU9PGbTJwhaOkgHD//ujL/fOIyyWiB/X1fDzOI891f859f859S7y9zn3fx2FZliUAAABckI+3GwAAALgWEJoAAAAMEJoAAAAMEJoAAAAMEJoAAAAMEJoAAAAMEJoAAAAMEJoAAAAMEJoAAAAMEJqAa0R4eLh69uzp7Taue+PHj1edOnXk6+urJk2aeLudi3I4HBo9erS32yhxH7Nnz5bD4dDGjRtLrZfRo0fL4XCU2vqAIoQmwAsu9ovi3nvv1e23337Z21myZEmZ+IV6rVi+fLmGDRum1q1ba9asWXrttde83ZIkjiNQVpTzdgMAzOzYsUM+Ppf275wlS5Zo6tSp/MI1tGLFCvn4+Oi9996Tn5+ft9uxXeg4njx5UuXK8b9y4Gpgpgm4Rvj7+6t8+fLebuOS5OTkeLuFS3Lw4EEFBgaWODD9+uuvpdzRxQUEBBCagKuE0ARcI84+pyk/P19jxoxR/fr1FRAQoGrVqqlNmzZKSUmRJPXs2VNTp06V9Nv5JkWPIjk5ORoyZIjCwsLk7++vBg0a6G9/+5ssy/LY7smTJzVw4EBVr15dlStX1h//+Ef9/PPPxc5hKTqP5IcfftDjjz+uKlWqqE2bNpKk7777Tj179lSdOnUUEBCgkJAQPfnkkzpy5IjHtorW8eOPP+qJJ56Qy+VSjRo19NJLL8myLO3fv18PPfSQnE6nQkJC9Oabbxrtu9OnT+uvf/2r6tatK39/f4WHh+v5559Xbm6uXeNwODRr1izl5OTY+2r27NnnXWfRR6jp6elq27atKlSooOeff17Sb+Grd+/eCg4OVkBAgBo3bqw5c+Z4vH7lypVyOBxauXKlx/iePXs8tn2x43i+47Br1y717NlTQUFBcrlc6tWrV7FQZ3psS2Lv3r36y1/+ogYNGigwMFDVqlXTo48+qj179pyz/tdff9VTTz2latWqyel06s9//rOOHTtWrG7p0qW6++67VbFiRVWuXFmdOnXS1q1bL6tXwBT/PAG8KDs7W4cPHy42np+ff9HXjh49WmPHjlWfPn3UsmVLud1ubdy4UZs2bdL999+vp556SgcOHFBKSoo++OADj9dalqU//vGP+uqrr9S7d281adJEy5Yt09ChQ/Xzzz/rrbfesmt79uypTz75RD169FCrVq20atUqderU6bx9Pfroo6pfv75ee+01O4ClpKToP//5j3r16qWQkBBt3bpVM2bM0NatW/XNN98UO2m3a9euatiwoV5//XUtXrxYr7zyiqpWrap3331Xv//97/XGG29o7ty5evbZZ9WiRQu1bdv2gvuqT58+mjNnjh555BENGTJE69at09ixY7Vt2zYtWLBAkvTBBx9oxowZWr9+vWbOnClJuuuuuy643iNHjuiBBx5QXFycnnjiCQUHB+vkyZO69957tWvXLiUkJCgiIkKffvqpevbsqaysLD3zzDMXXOfZLnQcL+Sxxx5TRESExo4dq02bNmnmzJmqWbOm3njjDbvmUo/tpdiwYYPWrl2ruLg43XzzzdqzZ4+mTZume++9Vz/88IMqVKjgUZ+QkKCgoCCNHj1aO3bs0LRp07R37147XEq/HaP4+HjFxsbqjTfe0K+//qpp06apTZs2+ve//63w8PBS6R04LwvAVTdr1ixL0gUft912m8drateubcXHx9vPGzdubHXq1OmC2xkwYIB1rh/zhQsXWpKsV155xWP8kUcesRwOh7Vr1y7LsiwrPT3dkmQNGjTIo65nz56WJGvUqFH22KhRoyxJVrdu3Ypt79dffy029tFHH1mSrNWrVxdbR79+/eyx06dPWzfffLPlcDis119/3R4/duyYFRgY6LFPzmXz5s2WJKtPnz4e488++6wlyVqxYoU9Fh8fb1WsWPGC6ytyzz33WJKs6dOne4xPnDjRkmR9+OGH9lheXp4VHR1tVapUyXK73ZZlWdZXX31lSbK++uorj9fv3r3bkmTNmjXLHjvfcbQs67zH4cknn/So+9Of/mRVq1bNfn4px9bE2a851zFPS0uzJFn/+Mc/7LGin4VmzZpZeXl59vi4ceMsSdbnn39uWZZlHT9+3AoKCrL69u3rsc6MjAzL5XJ5jBftA6C08fEc4EVTp05VSkpKsccdd9xx0dcGBQVp69at2rlz5yVvd8mSJfL19dXAgQM9xocMGSLLsrR06VJJUnJysiTpL3/5i0ddYmLiedf99NNPFxsLDAy0/3zq1CkdPnxYrVq1kiRt2rSpWH2fPn3sP/v6+qp58+ayLEu9e/e2x4OCgtSgQQP95z//OW8v0m/vVZKSkpI8xocMGSJJWrx48QVffyH+/v7q1atXse2FhISoW7du9lj58uU1cOBAnThxQqtWrSrx9i7F2cfh7rvv1pEjR+R2uyWV7NheijOPeX5+vo4cOaJ69eopKCjonMe8X79+Hufs9e/fX+XKlbOPX0pKirKystStWzcdPnzYfvj6+ioqKkpfffVVqfQNXAgfzwFe1LJlSzVv3rzYeJUqVc75sd2ZXn75ZT300EP63e9+p9tvv10dOnRQjx49jALX3r17FRoaqsqVK3uMN2zY0F5e9F8fHx9FRER41NWrV++86z67VpKOHj2qMWPG6OOPP9bBgwc9lmVnZxerv+WWWzyeu1wuBQQEqHr16sXGzz4v6mxF7+HsnkNCQhQUFGS/15K46aabip00vnfvXtWvX7/YlY5n79sr7ex9WKVKFUnSsWPH5HQ6S3RsL8XJkyc1duxYzZo1Sz///LPHuXLnOub169f3eF6pUiXVqlXLPgeq6B8Hv//978+5PafTWSp9AxdCaAKuUW3bttVPP/2kzz//XMuXL9fMmTP11ltvafr06R4zNVfbmTMMRR577DGtXbtWQ4cOVZMmTVSpUiUVFhaqQ4cOKiwsLFbv6+trNCap2Inr53MlbnZ4rvdq6nz9FBQUlHidZ7rc/XW5EhMTNWvWLA0aNEjR0dFyuVxyOByKi4s75zG/mKLXfPDBBwoJCSm2nCsIcTXwtwy4hlWtWlW9evVSr169dOLECbVt21ajR4+2Q9P5fjHXrl1bX375pY4fP+4x27R9+3Z7edF/CwsLtXv3bo+ZgF27dhn3eOzYMaWmpmrMmDEaOXKkPV6SjxVLoug97Ny5057tkaTMzExlZWXZ77U0t/fdd9+psLDQY7bp7H1bNPOTlZXl8fpzzURdicBXGsf2Qj777DPFx8d7XOF46tSpYu+3yM6dO9WuXTv7+YkTJ/TLL7+oY8eOkqS6detKkmrWrKmYmJhS6RG4VJzTBFyjzv5YqlKlSqpXr57HZfQVK1aUVPwXc8eOHVVQUKApU6Z4jL/11ltyOBx64IEHJEmxsbGSpHfeecejbvLkycZ9Fs14nD3DMXHiRON1XI6iX7pnb2/ChAmSVGpXi525vYyMDM2fP98eO336tCZPnqxKlSrpnnvukfRbaPH19dXq1as9Xn/2vpbOfxwvR2kc2wvx9fUtdswnT5583pm0GTNmeFw1Om3aNJ0+fdrj76LT6dRrr712zqtLDx06VCp9AxfCTBNwjYqMjNS9996rZs2aqWrVqtq4caM+++wzJSQk2DXNmjWTJA0cOFCxsbHy9fVVXFyc/vCHP6hdu3Z64YUXtGfPHjVu3FjLly/X559/rkGDBtn/qm/WrJm6dOmiiRMn6siRI/Zl6T/++KMksxkQp9Optm3baty4ccrPz9dNN92k5cuXa/fu3VdgrxTXuHFjxcfHa8aMGcrKytI999yj9evXa86cOercubPH7EZp6Nevn95991317NlT6enpCg8P12effaY1a9Zo4sSJ9syey+XSo48+qsmTJ8vhcKhu3bpatGhRsXO+pPMfx8tRGsf2Qh588EF98MEHcrlcioyMVFpamr788ktVq1btnPV5eXm677779Nhjj2nHjh1655131KZNG/3xj3+U9Nvfo2nTpqlHjx5q2rSp4uLiVKNGDe3bt0+LFy9W69ati/0jAChthCbgGjVw4EB98cUXWr58uXJzc1W7dm298sorGjp0qF3z8MMPKzExUR9//LE+/PBDWZaluLg4+fj46IsvvtDIkSM1f/58zZo1S+Hh4Ro/frx9VVmRf/zjHwoJCdFHH32kBQsWKCYmRvPnz1eDBg0UEBBg1Ou8efOUmJioqVOnyrIstW/fXkuXLlVoaGip7pPzmTlzpurUqaPZs2drwYIFCgkJ0YgRIzRq1KhS31ZgYKBWrlyp4cOHa86cOXK73WrQoIFmzZpV7AuXJ0+erPz8fE2fPl3+/v567LHHNH78+GLfO3i+43i5SuPYns/bb78tX19fzZ07V6dOnVLr1q315Zdf2jNcZ5syZYrmzp2rkSNHKj8/X926ddOkSZM8wtvjjz+u0NBQvf766xo/frxyc3N100036e677y52FSNwJTisq3VWIIDrxubNm3XnnXfqww8/VPfu3b3dDkoRxxY4P85pAnBBJ0+eLDY2ceJE+fj4XPRO3CjbOLbApeHjOQAXNG7cOKWnp6tdu3YqV66cli5dqqVLl6pfv34KCwvzdnu4DBc7tgUFBRc9wbpSpUqqVKnSVeoY8C4+ngNwQSkpKRozZox++OEHnThxQrfccot69OihF154gXvjXOMudmz37NlzzpuVnmnUqFGX/eW+wLWC0AQAOKdTp07pX//61wVr6tSpozp16lyljgDvIjQBAAAY4ERwAAAAA5yQUEoKCwt14MABVa5c+Yp85QEAACh9lmXp+PHjCg0NLfZF22cjNJWSAwcOcCURAADXqP379+vmm2++YA2hqZQUfTXC/v375XQ6vdwNAAAw4Xa7FRYW5vHl5edDaColRR/JOZ1OQhMAANcYk1NrOBEcAADAAKEJAADAAKEJAADAAKEJAADAAKEJAADAAKEJAADAAKEJAADAAKEJAADAAKEJAADAAKEJAADAAKEJAADAAKEJAADAAKEJAADAAKEJAADAAKEJAADAQDlvNwAA+E348MXebgEo0/a83smr22emCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwAChCQAAwIBXQ1NBQYFeeuklRUREKDAwUHXr1tVf//pXWZZl11iWpZEjR6pWrVoKDAxUTEyMdu7c6bGeo0ePqnv37nI6nQoKClLv3r114sQJj5rvvvtOd999twICAhQWFqZx48YV6+fTTz/VrbfeqoCAADVq1EhLliy5Mm8cAABcc7wamt544w1NmzZNU6ZM0bZt2/TGG29o3Lhxmjx5sl0zbtw4TZo0SdOnT9e6detUsWJFxcbG6tSpU3ZN9+7dtXXrVqWkpGjRokVavXq1+vXrZy93u91q3769ateurfT0dI0fP16jR4/WjBkz7Jq1a9eqW7du6t27t/7973+rc+fO6ty5s7Zs2XJ1dgYAACjTHNaZ0zpX2YMPPqjg4GC999579liXLl0UGBioDz/8UJZlKTQ0VEOGDNGzzz4rScrOzlZwcLBmz56tuLg4bdu2TZGRkdqwYYOaN28uSUpOTlbHjh313//+V6GhoZo2bZpeeOEFZWRkyM/PT5I0fPhwLVy4UNu3b5ckde3aVTk5OVq0aJHdS6tWrdSkSRNNnz79ou/F7XbL5XIpOztbTqez1PYRgBtH+PDF3m4BKNP2vN6p1Nd5Kb+/vTrTdNdddyk1NVU//vijJOnbb7/Vv/71Lz3wwAOSpN27dysjI0MxMTH2a1wul6KiopSWliZJSktLU1BQkB2YJCkmJkY+Pj5at26dXdO2bVs7MElSbGysduzYoWPHjtk1Z26nqKZoO2fLzc2V2+32eAAAgOtXOW9ufPjw4XK73br11lvl6+urgoICvfrqq+revbskKSMjQ5IUHBzs8brg4GB7WUZGhmrWrOmxvFy5cqpatapHTURERLF1FC2rUqWKMjIyLrids40dO1ZjxowpydsGAADXIK/ONH3yySeaO3eu5s2bp02bNmnOnDn629/+pjlz5nizLSMjRoxQdna2/di/f7+3WwIAAFeQV2eahg4dquHDhysuLk6S1KhRI+3du1djx45VfHy8QkJCJEmZmZmqVauW/brMzEw1adJEkhQSEqKDBw96rPf06dM6evSo/fqQkBBlZmZ61BQ9v1hN0fKz+fv7y9/fvyRvGwAAXIO8OtP066+/ysfHswVfX18VFhZKkiIiIhQSEqLU1FR7udvt1rp16xQdHS1Jio6OVlZWltLT0+2aFStWqLCwUFFRUXbN6tWrlZ+fb9ekpKSoQYMGqlKlil1z5naKaoq2AwAAbmxeDU1/+MMf9Oqrr2rx4sXas2ePFixYoAkTJuhPf/qTJMnhcGjQoEF65ZVX9MUXX+j777/Xn//8Z4WGhqpz586SpIYNG6pDhw7q27ev1q9frzVr1ighIUFxcXEKDQ2VJD3++OPy8/NT7969tXXrVs2fP19vv/22kpKS7F6eeeYZJScn680339T27ds1evRobdy4UQkJCVd9vwAAgLLHqx/PTZ48WS+99JL+8pe/6ODBgwoNDdVTTz2lkSNH2jXDhg1TTk6O+vXrp6ysLLVp00bJyckKCAiwa+bOnauEhATdd9998vHxUZcuXTRp0iR7ucvl0vLlyzVgwAA1a9ZM1atX18iRIz3u5XTXXXdp3rx5evHFF/X888+rfv36WrhwoW6//farszMAAECZ5tX7NF1PuE8TgMvFfZqAC7uh79MEAABwrSA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGPB6aPr555/1xBNPqFq1agoMDFSjRo20ceNGe7llWRo5cqRq1aqlwMBAxcTEaOfOnR7rOHr0qLp37y6n06mgoCD17t1bJ06c8Kj57rvvdPfddysgIEBhYWEaN25csV4+/fRT3XrrrQoICFCjRo20ZMmSK/OmAQDANceroenYsWNq3bq1ypcvr6VLl+qHH37Qm2++qSpVqtg148aN06RJkzR9+nStW7dOFStWVGxsrE6dOmXXdO/eXVu3blVKSooWLVqk1atXq1+/fvZyt9ut9u3bq3bt2kpPT9f48eM1evRozZgxw65Zu3atunXrpt69e+vf//63OnfurM6dO2vLli1XZ2cAAIAyzWFZluWtjQ8fPlxr1qzR119/fc7llmUpNDRUQ4YM0bPPPitJys7OVnBwsGbPnq24uDht27ZNkZGR2rBhg5o3by5JSk5OVseOHfXf//5XoaGhmjZtml544QVlZGTIz8/P3vbChQu1fft2SVLXrl2Vk5OjRYsW2dtv1aqVmjRpounTp1/0vbjdbrlcLmVnZ8vpdF7WfgFwYwofvtjbLQBl2p7XO5X6Oi/l97dXZ5q++OILNW/eXI8++qhq1qypO++8U3//+9/t5bt371ZGRoZiYmLsMZfLpaioKKWlpUmS0tLSFBQUZAcmSYqJiZGPj4/WrVtn17Rt29YOTJIUGxurHTt26NixY3bNmdspqinaztlyc3Pldrs9HgAA4Prl1dD0n//8R9OmTVP9+vW1bNky9e/fXwMHDtScOXMkSRkZGZKk4OBgj9cFBwfbyzIyMlSzZk2P5eXKlVPVqlU9as61jjO3cb6aouVnGzt2rFwul/0ICwu75PcPAACuHV4NTYWFhWratKlee+013XnnnerXr5/69u1r9HGYt40YMULZ2dn2Y//+/d5uCQAAXEFeDU21atVSZGSkx1jDhg21b98+SVJISIgkKTMz06MmMzPTXhYSEqKDBw96LD99+rSOHj3qUXOudZy5jfPVFC0/m7+/v5xOp8cDAABcv7wamlq3bq0dO3Z4jP3444+qXbu2JCkiIkIhISFKTU21l7vdbq1bt07R0dGSpOjoaGVlZSk9Pd2uWbFihQoLCxUVFWXXrF69Wvn5+XZNSkqKGjRoYF+pFx0d7bGdopqi7QAAgBubV0PT4MGD9c033+i1117Trl27NG/ePM2YMUMDBgyQJDkcDg0aNEivvPKKvvjiC33//ff685//rNDQUHXu3FnSbzNTHTp0UN++fbV+/XqtWbNGCQkJiouLU2hoqCTp8ccfl5+fn3r37q2tW7dq/vz5evvtt5WUlGT38swzzyg5OVlvvvmmtm/frtGjR2vjxo1KSEi46vsFAACUPeW8ufEWLVpowYIFGjFihF5++WVFRERo4sSJ6t69u10zbNgw5eTkqF+/fsrKylKbNm2UnJysgIAAu2bu3LlKSEjQfffdJx8fH3Xp0kWTJk2yl7tcLi1fvlwDBgxQs2bNVL16dY0cOdLjXk533XWX5s2bpxdffFHPP/+86tevr4ULF+r222+/OjsDAACUaV69T9P1hPs0Abhc3KcJuLAb+j5NAAAA1wpCEwAAgAFCEwAAgAFCEwAAgAFCEwAAgAFCEwAAgAFCEwAAgAFCEwAAgAFCEwAAgAFCEwAAgAFCEwAAgAFCEwAAgAFCEwAAgAFCEwAAgIEShaY6deroyJEjxcazsrJUp06dy24KAACgrClRaNqzZ48KCgqKjefm5urnn3++7KYAAADKmnKXUvzFF1/Yf162bJlcLpf9vKCgQKmpqQoPDy+15gAAAMqKSwpNnTt3liQ5HA7Fx8d7LCtfvrzCw8P15ptvllpzAAAAZcUlhabCwkJJUkREhDZs2KDq1atfkaYAAADKmksKTUV2795d2n0AAACUaSUKTZKUmpqq1NRUHTx40J6BKvL+++9fdmMAAABlSYlC05gxY/Tyyy+refPmqlWrlhwOR2n3BQAAUKaUKDRNnz5ds2fPVo8ePUq7HwAAgDKpRPdpysvL01133VXavQAAAJRZJQpNffr00bx580q7FwAAgDKrRB/PnTp1SjNmzNCXX36pO+64Q+XLl/dYPmHChFJpDgAAoKwoUWj67rvv1KRJE0nSli1bPJZxUjgAALgelSg0ffXVV6XdBwAAQJlWonOaAAAAbjQlmmlq167dBT+GW7FiRYkbAgAAKItKFJqKzmcqkp+fr82bN2vLli3FvsgXAADgelCi0PTWW2+dc3z06NE6ceLEZTUEAABQFpXqOU1PPPEE3zsHAACuS6UamtLS0hQQEFCaqwQAACgTSvTx3MMPP+zx3LIs/fLLL9q4caNeeumlUmkMAACgLClRaHK5XB7PfXx81KBBA7388stq3759qTQGAABQlpQoNM2aNau0+wAAACjTShSaiqSnp2vbtm2SpNtuu0133nlnqTQFAABQ1pQoNB08eFBxcXFauXKlgoKCJElZWVlq166dPv74Y9WoUaM0ewQAAPC6El09l5iYqOPHj2vr1q06evSojh49qi1btsjtdmvgwIGl3SMAAIDXlWimKTk5WV9++aUaNmxoj0VGRmrq1KmcCA4AAK5LJZppKiwsVPny5YuNly9fXoWFhZfdFAAAQFlTotD0+9//Xs8884wOHDhgj/38888aPHiw7rvvvlJrDgAAoKwoUWiaMmWK3G63wsPDVbduXdWtW1cRERFyu92aPHlyafcIAADgdSU6pyksLEybNm3Sl19+qe3bt0uSGjZsqJiYmFJtDgAAoKy4pJmmFStWKDIyUm63Ww6HQ/fff78SExOVmJioFi1a6LbbbtPXX399pXoFAADwmksKTRMnTlTfvn3ldDqLLXO5XHrqqac0YcKEUmsOAACgrLik0PTtt9+qQ4cO513evn17paenX3ZTAAAAZc0lhabMzMxz3mqgSLly5XTo0KHLbgoAAKCsuaTQdNNNN2nLli3nXf7dd9+pVq1al90UAABAWXNJoaljx4566aWXdOrUqWLLTp48qVGjRunBBx8steYAAADKiku65cCLL76o//3f/9Xvfvc7JSQkqEGDBpKk7du3a+rUqSooKNALL7xwRRoFAADwpksKTcHBwVq7dq369++vESNGyLIsSZLD4VBsbKymTp2q4ODgK9IoAACAN13yzS1r166tJUuW6NixY9q1a5csy1L9+vVVpUqVK9EfAABAmVCiO4JLUpUqVdSiRYvS7AUAAKDMKtF3zwEAANxoCE0AAAAGCE0AAAAGCE0AAAAGCE0AAAAGCE0AAAAGCE0AAAAGCE0AAAAGykxoev311+VwODRo0CB77NSpUxowYICqVaumSpUqqUuXLsrMzPR43b59+9SpUydVqFBBNWvW1NChQ3X69GmPmpUrV6pp06by9/dXvXr1NHv27GLbnzp1qsLDwxUQEKCoqCitX7/+SrxNAABwjSoToWnDhg169913dccdd3iMDx48WP/85z/16aefatWqVTpw4IAefvhhe3lBQYE6deqkvLw8rV27VnPmzNHs2bM1cuRIu2b37t3q1KmT2rVrp82bN2vQoEHq06ePli1bZtfMnz9fSUlJGjVqlDZt2qTGjRsrNjZWBw8evPJvHgAAXBMcVtG37nrJiRMn1LRpU73zzjt65ZVX1KRJE02cOFHZ2dmqUaOG5s2bp0ceeUSStH37djVs2FBpaWlq1aqVli5dqgcffFAHDhywvyh4+vTpeu6553To0CH5+fnpueee0+LFi7VlyxZ7m3FxccrKylJycrIkKSoqSi1atNCUKVMkSYWFhQoLC1NiYqKGDx9u9D7cbrdcLpeys7PldDpLcxcBuEGED1/s7RaAMm3P651KfZ2X8vvb6zNNAwYMUKdOnRQTE+Mxnp6ervz8fI/xW2+9VbfccovS0tIkSWlpaWrUqJEdmCQpNjZWbrdbW7dutWvOXndsbKy9jry8PKWnp3vU+Pj4KCYmxq45l9zcXLndbo8HAAC4fpX4C3tLw8cff6xNmzZpw4YNxZZlZGTIz89PQUFBHuPBwcHKyMiwa84MTEXLi5ZdqMbtduvkyZM6duyYCgoKzlmzffv28/Y+duxYjRkzxuyNAgCAa57XZpr279+vZ555RnPnzlVAQIC32iixESNGKDs7237s37/f2y0BAIAryGuhKT09XQcPHlTTpk1Vrlw5lStXTqtWrdKkSZNUrlw5BQcHKy8vT1lZWR6vy8zMVEhIiCQpJCSk2NV0Rc8vVuN0OhUYGKjq1avL19f3nDVF6zgXf39/OZ1OjwcAALh+eS003Xffffr++++1efNm+9G8eXN1797d/nP58uWVmppqv2bHjh3at2+foqOjJUnR0dH6/vvvPa5yS0lJkdPpVGRkpF1z5jqKaorW4efnp2bNmnnUFBYWKjU11a4BAADw2jlNlStX1u233+4xVrFiRVWrVs0e7927t5KSklS1alU5nU4lJiYqOjparVq1kiS1b99ekZGR6tGjh8aNG6eMjAy9+OKLGjBggPz9/SVJTz/9tKZMmaJhw4bpySef1IoVK/TJJ59o8eL/f5VKUlKS4uPj1bx5c7Vs2VITJ05UTk6OevXqdZX2BgAAKOu8eiL4xbz11lvy8fFRly5dlJubq9jYWL3zzjv2cl9fXy1atEj9+/dXdHS0KlasqPj4eL388st2TUREhBYvXqzBgwfr7bff1s0336yZM2cqNjbWrunatasOHTqkkSNHKiMjQ02aNFFycnKxk8MBAMCNy+v3abpecJ8mAJeL+zQBF3bD36cJAADgWkBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMEBoAgAAMODV0DR27Fi1aNFClStXVs2aNdW5c2ft2LHDo+bUqVMaMGCAqlWrpkqVKqlLly7KzMz0qNm3b586deqkChUqqGbNmho6dKhOnz7tUbNy5Uo1bdpU/v7+qlevnmbPnl2sn6lTpyo8PFwBAQGKiorS+vXrS/09AwCAa5NXQ9OqVas0YMAAffPNN0pJSVF+fr7at2+vnJwcu2bw4MH65z//qU8//VSrVq3SgQMH9PDDD9vLCwoK1KlTJ+Xl5Wnt2rWaM2eOZs+erZEjR9o1u3fvVqdOndSuXTtt3rxZgwYNUp8+fbRs2TK7Zv78+UpKStKoUaO0adMmNW7cWLGxsTp48ODV2RkAAKBMc1iWZXm7iSKHDh1SzZo1tWrVKrVt21bZ2dmqUaOG5s2bp0ceeUSStH37djVs2FBpaWlq1aqVli5dqgcffFAHDhxQcHCwJGn69Ol67rnndOjQIfn5+em5557T4sWLtWXLFntbcXFxysrKUnJysiQpKipKLVq00JQpUyRJhYWFCgsLU2JiooYPH37R3t1ut1wul7Kzs+V0Okt71wC4AYQPX+ztFoAybc/rnUp9nZfy+7tMndOUnZ0tSapataokKT09Xfn5+YqJibFrbr31Vt1yyy1KS0uTJKWlpalRo0Z2YJKk2NhYud1ubd261a45cx1FNUXryMvLU3p6ukeNj4+PYmJi7Jqz5ebmyu12ezwAAMD1q8yEpsLCQg0aNEitW7fW7bffLknKyMiQn5+fgoKCPGqDg4OVkZFh15wZmIqWFy27UI3b7dbJkyd1+PBhFRQUnLOmaB1nGzt2rFwul/0ICwsr2RsHAADXhDITmgYMGKAtW7bo448/9nYrRkaMGKHs7Gz7sX//fm+3BAAArqBy3m5AkhISErRo0SKtXr1aN998sz0eEhKivLw8ZWVlecw2ZWZmKiQkxK45+yq3oqvrzqw5+4q7zMxMOZ1OBQYGytfXV76+vuesKVrH2fz9/eXv71+yNwwAAK45Xp1psixLCQkJWrBggVasWKGIiAiP5c2aNVP58uWVmppqj+3YsUP79u1TdHS0JCk6Olrff/+9x1VuKSkpcjqdioyMtGvOXEdRTdE6/Pz81KxZM4+awsJCpaam2jUAAODG5tWZpgEDBmjevHn6/PPPVblyZfv8IZfLpcDAQLlcLvXu3VtJSUmqWrWqnE6nEhMTFR0drVatWkmS2rdvr8jISPXo0UPjxo1TRkaGXnzxRQ0YMMCeCXr66ac1ZcoUDRs2TE8++aRWrFihTz75RIsX//8rVZKSkhQfH6/mzZurZcuWmjhxonJyctSrV6+rv2MAAECZ49XQNG3aNEnSvffe6zE+a9Ys9ezZU5L01ltvycfHR126dFFubq5iY2P1zjvv2LW+vr5atGiR+vfvr+joaFWsWFHx8fF6+eWX7ZqIiAgtXrxYgwcP1ttvv62bb75ZM2fOVGxsrF3TtWtXHTp0SCNHjlRGRoaaNGmi5OTkYieHAwCAG1OZuk/TtYz7NAG4XNynCbgw7tMEAABwDSA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCA0AQAAGCjn7QZghm8/B87vSnzzOQCcjZkmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4QmAAAAA4Sms0ydOlXh4eEKCAhQVFSU1q9f7+2WAABAGUBoOsP8+fOVlJSkUaNGadOmTWrcuLFiY2N18OBBb7cGAAC8jNB0hgkTJqhv377q1auXIiMjNX36dFWoUEHvv/++t1sDAABeRmj6P3l5eUpPT1dMTIw95uPjo5iYGKWlpXmxMwAAUBaU83YDZcXhw4dVUFCg4OBgj/Hg4GBt3769WH1ubq5yc3Pt59nZ2ZIkt9t9RforzP31iqwXuB5cqZ+7q42fc+DCrsTPetE6Lcu6aC2hqYTGjh2rMWPGFBsPCwvzQjfAjc010dsdALgaruTP+vHjx+VyuS5YQ2j6P9WrV5evr68yMzM9xjMzMxUSElKsfsSIEUpKSrKfFxYW6ujRo6pWrZocDscV7xfe43a7FRYWpv3798vpdHq7HQBXAD/nNw7LsnT8+HGFhoZetJbQ9H/8/PzUrFkzpaamqnPnzpJ+C0KpqalKSEgoVu/v7y9/f3+PsaCgoKvQKcoKp9PJ/0yB6xw/5zeGi80wFSE0nSEpKUnx8fFq3ry5WrZsqYkTJyonJ0e9evXydmsAAMDLCE1n6Nq1qw4dOqSRI0cqIyNDTZo0UXJycrGTwwEAwI2H0HSWhISEc34cBxTx9/fXqFGjin08C+D6wc85zsVhmVxjBwAAcIPj5pYAAAAGCE0AAAAGCE0AAAAGCE0AAAAGCE3AJZg6darCw8MVEBCgqKgorV+/3tstAShlq1ev1h/+8AeFhobK4XBo4cKF3m4JZQShCTA0f/58JSUladSoUdq0aZMaN26s2NhYHTx40NutAShFOTk5aty4saZOnertVlDGcMsBwFBUVJRatGihKVOmSPrta3bCwsKUmJio4cOHe7k7AFeCw+HQggUL7K/Xwo2NmSbAQF5entLT0xUTE2OP+fj4KCYmRmlpaV7sDABwtRCaAAOHDx9WQUFBsa/UCQ4OVkZGhpe6AgBcTYQmAAAAA4QmwED16tXl6+urzMxMj/HMzEyFhIR4qSsAwNVEaAIM+Pn5qVmzZkpNTbXHCgsLlZqaqujoaC92BgC4Wsp5uwHgWpGUlKT4+Hg1b95cLVu21MSJE5WTk6NevXp5uzUApejEiRPatWuX/Xz37t3avHmzqlatqltuucWLncHbuOUAcAmmTJmi8ePHKyMjQ02aNNGkSZMUFRXl7bYAlKKVK1eqXbt2xcbj4+M1e/bsq98QygxCEwAAgAHOaQIAADBAaAIAADBAaAIAADBAaAIAADBAaAIAADBAaAIAADBAaAIAADBAaAJwQ3E4HFq4cOFV297o0aPVpEmTS3pNafTYs2dPde7c+bLWAcAToQnAVdezZ085HA45HA6VL19eERERGjZsmE6dOlVq2zhfWPnll1/0wAMPlNp2ANw4+O45AF7RoUMHzZo1S/n5+UpPT1d8fLwcDofeeOONK7rdkJCQK7p+ANcvZpoAeIW/v79CQkIUFhamzp07KyYmRikpKZKk3NxcDRw4UDVr1lRAQIDatGmjDRs22K+dPXu2goKCPNa3cOFCORwOe/mYMWP07bff2jNaRd8ZduZHX3v27JHD4dD//u//ql27dqpQoYIaN26stLQ0j3X//e9/V1hYmCpUqKA//elPmjBhQrHtm9qwYYPuv/9+Va9eXS6XS/fcc482bdpUrK5oRiwwMFB16tTRZ5995rF8//79euyxxxQUFKSqVavqoYce0p49e0rUEwAzhCYAXrdlyxatXbtWfn5+kqRhw4bpf/7nfzRnzhxt2rRJ9erVU2xsrI4ePWq0vq5du2rIkCG67bbb9Msvv+iXX35R165dz1v/wgsv6Nlnn9XmzZv1u9/9Tt26ddPp06clSWvWrNHTTz+tZ555Rps3b9b999+vV199tcTv9fjx44qPj9e//vUvffPNN6pfv746duyo48ePe9S99NJL6tKli7799lt1795dcXFx2rZtmyQpPz9fsbGxqly5sr7++mutWbNGlSpVUocOHZSXl1fi3gBchAUAV1l8fLzl6+trVaxY0fL397ckWT4+PtZnn31mnThxwipfvrw1d+5cuz4vL88KDQ21xo0bZ1mWZc2aNctyuVwe61ywYIF15v/SRo0aZTVu3LjYtiVZCxYssCzLsnbv3m1JsmbOnGkv37p1qyXJ2rZtm2VZltW1a1erU6dOHuvo3r17se2fz/n6KFJQUGBVrlzZ+uc//+nR49NPP+1RFxUVZfXv39+yLMv64IMPrAYNGliFhYX28tzcXCswMNBatmyZZVm/7eOHHnrIqEcAZphpAuAV7dq10+bNm7Vu3TrFx8erV69e6tKli3766Sfl5+erdevWdm358uXVsmVLe6altN1xxx32n2vVqiVJOnjwoCRpx44datmypUf92c8vRWZmpvr27av69evL5XLJ6XTqxIkT2rdvn0dddHR0sedF7//bb7/Vrl27VLlyZVWqVEmVKlVS1apVderUKf30008l7g3AhXEiOACvqFixourVqydJev/999W4cWO99957atGixUVf6+PjI8uyPMby8/NL3Ev58uXtPxedF1VYWFji9V1IfHy8jhw5orffflu1a9eWv7+/oqOjL+ljtRMnTqhZs2aaO3dusWU1atQozXYBnIGZJgBe5+Pjo+eff14vvvii6tatKz8/P61Zs8Zenp+frw0bNigyMlLSb8Hg+PHjysnJsWs2b97ssU4/Pz8VFBRcdm8NGjTwOAldUrHnl2LNmjUaOHCgOnbsqNtuu03+/v46fPhwsbpvvvmm2POGDRtKkpo2baqdO3eqZs2aqlevnsfD5XKVuDcAF0ZoAlAmPProo/L19dW0adPUv39/DR06VMnJyfrhhx/Ut29f/frrr+rdu7ckKSoqShUqVNDzzz+vn376SfPmzbOvjisSHh6u3bt3a/PmzTp8+LByc3NL1FdiYqKWLFmiCRMmaOfOnXr33Xe1dOlSe0bqUtWvX18ffPCBtm3bpnXr1ql79+4KDAwsVvfpp5/q/fff148//qhRo0Zp/fr1SkhIkCR1795d1atX10MPPaSvv/5au3fv1sqVKzVw4ED997//LVFfAC6O0ASgTChXrpwSEhI0btw4vfrqq+rSpYt69Oihpk2bateuXVq2bJmqVKkiSapatao+/PBDLVmyRI0aNdJHH32k0aNHe6yvS5cu6tChg9q1a6caNWroo48+KlFfrVu31vTp0zVhwgQ1btxYycnJGjx4sAICAkq0vvfee0/Hjh1T06ZN1aNHD/vWCmcbM2aMPv74Y91xxx36xz/+oY8++sieaatQoYJWr16tW265RQ8//LAaNmyo3r1769SpU3I6nSXqC8DFOayzTwwAAFxQ3759tX37dn399dfebgXAVcSJ4ABwEX/72990//33q2LFilq6dKnmzJmjd955x9ttAbjKmGkCgIt47LHHtHLlSh0/flx16tRRYmKinn76aUnSbbfdpr17957zde+++666d+9+NVsFcAURmgDgMuzdu/e8tzsIDg5W5cqVr3JHAK4UQhMAAIABrp4DAAAwQGgCAAAwQGgCAAAwQGgCAAAwQGgCAAAwQGgCAAAwQGgCAAAwQGgCAAAw8P8AHuxB5agbuaMAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_df[\"routing_label\"] = train_df[\"mixtral_score\"].apply(\n",
    "    lambda x: 1 if x >= 4 else 0\n",
    ")\n",
    "\n",
    "visualize_label_distribution(train_df, key=\"routing_label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2: Finetune a router model <a id=\"finetune-router-model\"></a>\n",
    "\n",
    "In this section, we will explain how to finetune a causal LLM classifier to be an effective router. While our data contains `gpt4_response` and `mixtral_response`, we will only use the pair (`query`, `mixtral_score`) for training. The goal is for the router to rely solely on the query text to determine which model to route to. Our approach is straightforward: we train a 5-way classifier to predict the `mixtral_score` from the `query`. At inference time, we will route to Mixtral if our router predicts a high score (i.e., 4-5) and to GPT-4 otherwise.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Data Preparation\n",
    "We will discuss a few preprocessing steps to prepare the data for finetuning an LLM classifier.\n",
    "\n",
    "### Task Instructions\n",
    "We use the instruction-following framework to finetune an LLM as a router. The task instructions guide the model to predict the score label for a given query. They ensure the model understands the evaluation criteria and can accurately assess the query's complexity and expected response quality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Instruction]\n",
      "Based on the question provided below, predict the score an expert evaluator would give to an AI assistant's response, considering its helpfulness, relevance, adherence to facts, depth, creativity, and detail. Your prediction should infer the level of proficiency needed to address the question effectively. Use a scale from 1 to 5, where a higher score indicates a higher anticipated quality of response. Provide your prediction as: \"[[predicted rating]]\".\n",
      "\n",
      "Score criteria:\n",
      "- **4-5**: The AI assistant can produce a very strong answer, showing deep understanding, creativity, detailed insight, and high relevance.\n",
      "- **3**: The AI assistant can provide an adequate answer with moderate detail, relevance, and factual accuracy.\n",
      "- **1-2**: The AI assistant will struggle to produce a strong answer due to the question's difficulty, vagueness, or the assistant's limitations.\n",
      "\n",
      "[Question]\n",
      "{question}\n",
      "\n",
      "Prediction:\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from src.utils import inspect_instructions\n",
    "\n",
    "inspect_instructions()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### API Data Format\n",
    "\n",
    "To finetune the model, we must format the data to be compatible with [Anyscale's finetuning API](https://docs.anyscale.com/endpoints/fine-tuning/dataset-prep).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'role': 'system',\n",
       "  'content': '[Instruction]\\nBased on the question provided below, predict the score an expert evaluator would give to an AI assistant\\'s response, considering its helpfulness, relevance, adherence to facts, depth, creativity, and detail. Your prediction should infer the level of proficiency needed to address the question effectively. Use a scale from 1 to 5, where a higher score indicates a higher anticipated quality of response. Provide your prediction as: \"[[predicted rating]]\".\\n\\nScore criteria:\\n- **4-5**: The AI assistant can produce a very strong answer, showing deep understanding, creativity, detailed insight, and high relevance.\\n- **3**: The AI assistant can provide an adequate answer with moderate detail, relevance, and factual accuracy.\\n- **1-2**: The AI assistant will struggle to produce a strong answer due to the question\\'s difficulty, vagueness, or the assistant\\'s limitations.\\n'},\n",
       " {'role': 'user',\n",
       "  'content': \"[Question]\\nI'll give you a review, can you extract the food aspects and the opinion words of these aspects and analyze the sentiment of these opinion from this review? the review is:They tore the old NAME_1 down then built another one...? Anyway, they sell wine and beer and snacks and have a seating area inside and outside to eat. Besides gas, the big draw is the Made to Order food. I ordered some tacos and French toast sticks both were pretty good. I think I'd like to try more snacks.And they're open 24/7.\\n\\nPrediction:\\n\"},\n",
       " {'role': 'assistant', 'content': '[[4]]'}]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from src.utils import prepare_ft_messages\n",
    "\n",
    "train_df[\"messages\"] = prepare_ft_messages(train_df, \"mixtral_score\")\n",
    "\n",
    "# here's what the API data format looks like:\n",
    "display(train_df[\"messages\"].iloc[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Label Rebalancing\n",
    "\n",
    "For classification tasks, it's recommended to train on label-balanced datasets to ensure models are not biased to a specific label. We will balance the dataset based on `routing_label`, as this is the label of primary interest.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train size: 29504\n"
     ]
    }
   ],
   "source": [
    "from src.utils import balance_dataset\n",
    "\n",
    "balanced_train_df = balance_dataset(train_df, key=\"routing_label\")\n",
    "\n",
    "print(f\"Train size: {len(balanced_train_df)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Subsample and Store Data\n",
    "\n",
    "To expedite the time to run this tutorial, we will subsample 1,000 examples for training. We'll store the data in JSONL format to prepare for launching the finetuning job in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_sample = 1000\n",
    "output_file = \"/mnt/user_storage/train_data_sample.jsonl\"\n",
    "\n",
    "subsampled_df = balanced_train_df.sample(n=n_sample, random_state=42)\n",
    "subsampled_df.to_json(output_file, orient=\"records\", lines=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Fine-tune with Anyscale API\n",
    "\n",
    "We will run a fine-tuning job using Anyscale's LLM finetuning API as an isolated job, similar to our [end-to-end LLM workflows guide](https://github.com/anyscale/e2e-llm-workflows?tab=readme-ov-file#fine-tuning-1).\n",
    "\n",
    "For this tutorial, we will perform full-parameter finetuning of Llama3-8B on the same 1,000 samples we showed earlier to debug the training dynamics and ensure the model can fit the training set. Below, we present the training and job configurations before submitting the training job.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model_id: meta-llama/Meta-Llama-3-8B\n",
      "train_path: /mnt/user_storage/train_data_sample.jsonl\n",
      "valid_path: /mnt/user_storage/train_data_sample.jsonl\n",
      "context_length: 1024\n",
      "num_devices: 8\n",
      "num_epochs: 5\n",
      "checkpoint_every_n_epochs: 5\n",
      "train_batch_size_per_device: 4\n",
      "eval_batch_size_per_device: 4\n",
      "lr_scheduler_type: constant\n",
      "learning_rate: 1e-5\n",
      "num_checkpoints_to_keep: 1\n",
      "no_gradient_checkpoint: False\n",
      "output_dir: /mnt/local_storage\n",
      "deepspeed:\n",
      "  config_path: config_files/deepspeed/zero_3_optimizer_parameter_offload.json\n",
      "flash_attention_2: true\n",
      "classifier_config:\n",
      "  label_tokens:\n",
      "      - \"[[1]]\"\n",
      "      - \"[[2]]\"\n",
      "      - \"[[3]]\"\n",
      "      - \"[[4]]\"\n",
      "      - \"[[5]]\"\n"
     ]
    }
   ],
   "source": [
    "# View the full-param finetuning configuration for llama-3-8B\n",
    "!cat configs/ft_config_a10.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name: llm-router-tutorial\n",
      "entrypoint: python src/ft.py configs/ft_config_a10.yaml\n",
      "image_uri: localhost:5555/anyscale/llm-forge:0.5.0.0\n",
      "requirements: requirements.txt\n",
      "max_retries: 0\n"
     ]
    }
   ],
   "source": [
    "# View job yaml config\n",
    "!cat configs/ft_job.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[36mOutput\u001b[0m\u001b[0m\n",
      "\u001b[0m\u001b[1m\u001b[36m(anyscale +1.0s)\u001b[0m \u001b[0m\u001b[0m\u001b[0m\u001b[0mSubmitting job with config JobConfig(name='llm-router-tutorial', image_uri='localhost:5555/anyscale/llm-forge:0.5.0.0', compute_config=None, env_vars=None, py_modules=None, cloud=None, project=None, ray_version=None).\u001b[0m\n",
      "\u001b[0m\u001b[1m\u001b[36m(anyscale +2.1s)\u001b[0m \u001b[0m\u001b[0m\u001b[0m\u001b[0mUploading local dir '.' to cloud storage.\u001b[0m\n",
      "\u001b[0m\u001b[1m\u001b[36m(anyscale +3.2s)\u001b[0m \u001b[0m\u001b[0m\u001b[0m\u001b[0mJob 'llm-router-tutorial' submitted, ID: 'prodjob_cj7ek1pc72672hq1htqg17ca15'.\u001b[0m\n",
      "\u001b[0m\u001b[1m\u001b[36m(anyscale +3.2s)\u001b[0m \u001b[0m\u001b[0m\u001b[0m\u001b[0mView the job in the UI: https://console.anyscale.com/jobs/prodjob_cj7ek1pc72672hq1htqg17ca15\u001b[0m\n",
      "\u001b[0m\u001b[1m\u001b[36m(anyscale +3.2s)\u001b[0m \u001b[0m\u001b[0m\u001b[0m\u001b[0mUse `--wait` to wait for the job to run and stream logs.\u001b[0m\n",
      "\u001b[0m\u001b[0m"
     ]
    }
   ],
   "source": [
    "# Job submission\n",
    "!anyscale job submit --config-file configs/ft_job.yaml --exclude assets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The job takes around 10 minutes on `4xA100-80gb` and 1 hour on `8xA10-22gb` to finish. Training logs will show the final model checkpoint, e.g.:\n",
    "\n",
    "```\n",
    "Best checkpoint is stored in:\n",
    "storage-bucket-cld-tffbxe9ia5phqr1unxhz4f7e1e/org_4snvy99zwbmh4gbtk64jfqggmj/cld_tffbxe9ia5phqr1unxhz4f7e1e/artifact_storage/amjad__almahairi_dkaubsimoyxpiksqxqkxrfgfvzzotwtacs/llmforge-finetuning/meta-llama/Meta-Llama-3-8B/TorchTrainer_2024-06-21_17-02-52/epoch-4\n",
    "With perplexity: 1.0318867739521242\n",
    "```\n",
    "This checkpoint can be used to run batch inference or serve the model online."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3: Offline Evaluation <a id=\"offline-eval\"></a>\n",
    "\n",
    "Next, we will conduct an offline evaluation of the model trained on an out-of-domain dataset. The same model, now trained on the full dataset, is available in the following GitHub repository: [https://github.com/lm-sys/RouteLLM/](https://github.com/lm-sys/RouteLLM/), along with other router models.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install `RouteLLM` package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clone the repository under /home/ray/default/\n",
    "!git clone https://github.com/lm-sys/RouteLLM.git /home/ray/default/RouteLLM\n",
    "\n",
    "# Change to the cloned repository directory\n",
    "%cd /home/ray/default/RouteLLM\n",
    "\n",
    "# Install the package with the specified extras\n",
    "!pip install -e .[eval]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference Example\n",
    "Let's show an example of loading the model and running inference with a single example sampled from our data. Note that you need to get access to `meta-llama/Meta-Llama-3-8B` in order to run these evaluations.  Let's first show how a formatted input looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt: What challenges did FDR face while in office\n",
      "Label: 5\n",
      "Messages:\n",
      "[{'content': '[Instruction]\\n'\n",
      "             'Based on the question provided below, predict the score an '\n",
      "             \"expert evaluator would give to an AI assistant's response, \"\n",
      "             'considering its helpfulness, relevance, adherence to facts, '\n",
      "             'depth, creativity, and detail. Your prediction should infer the '\n",
      "             'level of proficiency needed to address the question effectively. '\n",
      "             'Use a scale from 1 to 5, where a higher score indicates a higher '\n",
      "             'anticipated quality of response. Provide your prediction as: '\n",
      "             '\"[[predicted rating]]\".\\n'\n",
      "             '\\n'\n",
      "             'Score criteria:\\n'\n",
      "             '- **4-5**: The AI assistant can produce a very strong answer, '\n",
      "             'showing deep understanding, creativity, detailed insight, and '\n",
      "             'high relevance.\\n'\n",
      "             '- **3**: The AI assistant can provide an adequate answer with '\n",
      "             'moderate detail, relevance, and factual accuracy.\\n'\n",
      "             '- **1-2**: The AI assistant will struggle to produce a strong '\n",
      "             \"answer due to the question's difficulty, vagueness, or the \"\n",
      "             \"assistant's limitations.\\n\",\n",
      "  'role': 'system'},\n",
      " {'content': '[Question]\\n'\n",
      "             'What challenges did FDR face while in office\\n'\n",
      "             '\\n'\n",
      "             'Prediction:\\n',\n",
      "  'role': 'user'},\n",
      " {'content': '[[5]]', 'role': 'assistant'}]\n"
     ]
    }
   ],
   "source": [
    "# Store your `meta-llama` access token in /home/ray/default/.env with the name LLAMA2_HF_TOKEN\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"/home/ray/default/.env\")\n",
    "\n",
    "from pprint import pprint\n",
    "\n",
    "# Sample one row from the DataFrame\n",
    "sampled_row = train_df.sample(n=1, random_state=42)\n",
    "\n",
    "# Convert the sampled row to a dictionary without the index\n",
    "input_example = sampled_row.to_dict(orient='records')[0]\n",
    "\n",
    "print(\"Prompt:\", input_example['prompt'])\n",
    "print(\"Label:\", input_example['mixtral_score'])\n",
    "print(\"Messages:\")\n",
    "pprint(input_example['messages'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run inference with this example and examine the model's output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading model checkpoint from routellm/causal_llm_gpt4_augmented ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading checkpoint shards: 100%|██████████| 4/4 [00:02<00:00,  1.76it/s]\n",
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done loading model in 5.628355264663696 seconds.\n",
      "{'binary_prob': 0.9662781,\n",
      " 'output_ids': tensor([128006,  78191, 128007,    271, 128260, 128009]),\n",
      " 'output_str': '<|start_header_id|>assistant<|end_header_id|>\\n'\n",
      "               '\\n'\n",
      "               '[[5]]<|eot_id|>',\n",
      " 'output_tokens': ['<|start_header_id|>',\n",
      "                   'assistant',\n",
      "                   '<|end_header_id|>',\n",
      "                   'ĊĊ',\n",
      "                   '[[5]]',\n",
      "                   '<|eot_id|>'],\n",
      " 'score_logits': array([10.3125, 10.9375, 11.4375, 14.4375, 15.    ], dtype=float32),\n",
      " 'score_pred': 5,\n",
      " 'softmax_scores': array([0.00566901, 0.0105911 , 0.01746178, 0.3507292 , 0.6155489 ],\n",
      "      dtype=float32)}\n"
     ]
    }
   ],
   "source": [
    "from src.offline_inference import single_example_inference\n",
    "\n",
    "result = single_example_inference(input_example)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model outputs the predicted score as a special token`[[5]]`, since it is trained to predict one of the 5 labels which we add as special tokens to the vocabulary. We extract softmax scores of each of 5 labels in `softmax_scores`, and compute the routing probability as `binary_prob = sum(softmax_scores[3:])`.\n",
    "\n",
    "To optimize inference speed, we can append the header tokens `<|start_header_id|>assistant<|end_header_id|>\\n\\n`  so the first token that the model outputs is the predicted label."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Benchmark Evaluation\n",
    "We will use the RouteLLM evaluation framework to measure the performance of our router against a random router on GSM8K. \n",
    "We report the percentage of calls the router needs to send to GPT-4 in order to achieve `20%`, `50%` and `80%` of GPT-4 performance, along with area under curve. \n",
    "See our [paper](https://arxiv.org/pdf/2406.18665) for more details on the evalaution metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Namespace(routers=['random', 'causal_llm'], benchmark='gsm8k', output='.', overwrite_cache=[], parallel=96, config='config.example.yaml', num_results=10)\n",
      "INFO: Pandarallel will run on 96 workers.\n",
      "INFO: Pandarallel will use Memory file system to transfer data between the main process and workers.\n",
      "Running eval for GSM8k.\n",
      "1307/1319 questions for GSM8K after decontamination.\n",
      "Evaluating router: random with threshold 0.00024069652516689466...\n",
      "\n",
      "=============== random with threshold 0.00024069652516689466 on gsm8k ===============\n",
      "Average accuracy: 85.769\n",
      "Model counts: gpt-4-1106-preview: 1307\n",
      "Model %: gpt-4-1106-preview: 100.000%\n",
      "=====================================================================================\n",
      "\n",
      "Evaluating router: random with threshold 0.07792316246593108...\n",
      "\n",
      "=============== random with threshold 0.07792316246593108 on gsm8k ===============\n",
      "Average accuracy: 83.168\n",
      "Model counts: gpt-4-1106-preview: 1176, mistralai/Mixtral-8x7B-Instruct-v0.1: 131\n",
      "Model %: gpt-4-1106-preview: 89.977%, mistralai/Mixtral-8x7B-Instruct-v0.1: 10.023%\n",
      "==================================================================================\n",
      "\n",
      "Evaluating router: random with threshold 0.18951071797526106...\n",
      "\n",
      "=============== random with threshold 0.18951071797526106 on gsm8k ===============\n",
      "Average accuracy: 80.566\n",
      "Model counts: gpt-4-1106-preview: 1045, mistralai/Mixtral-8x7B-Instruct-v0.1: 262\n",
      "Model %: gpt-4-1106-preview: 79.954%, mistralai/Mixtral-8x7B-Instruct-v0.1: 20.046%\n",
      "==================================================================================\n",
      "\n",
      "Evaluating router: random with threshold 0.3032111564929487...\n",
      "\n",
      "=============== random with threshold 0.3032111564929487 on gsm8k ===============\n",
      "Average accuracy: 78.653\n",
      "Model counts: gpt-4-1106-preview: 915, mistralai/Mixtral-8x7B-Instruct-v0.1: 392\n",
      "Model %: gpt-4-1106-preview: 70.008%, mistralai/Mixtral-8x7B-Instruct-v0.1: 29.992%\n",
      "=================================================================================\n",
      "\n",
      "Evaluating router: random with threshold 0.4032666385017831...\n",
      "\n",
      "=============== random with threshold 0.4032666385017831 on gsm8k ===============\n",
      "Average accuracy: 76.511\n",
      "Model counts: gpt-4-1106-preview: 784, mistralai/Mixtral-8x7B-Instruct-v0.1: 523\n",
      "Model %: gpt-4-1106-preview: 59.985%, mistralai/Mixtral-8x7B-Instruct-v0.1: 40.015%\n",
      "=================================================================================\n",
      "\n",
      "Evaluating router: random with threshold 0.49540592491118873...\n",
      "\n",
      "=============== random with threshold 0.49540592491118873 on gsm8k ===============\n",
      "Average accuracy: 73.986\n",
      "Model counts: gpt-4-1106-preview: 654, mistralai/Mixtral-8x7B-Instruct-v0.1: 653\n",
      "Model %: gpt-4-1106-preview: 50.038%, mistralai/Mixtral-8x7B-Instruct-v0.1: 49.962%\n",
      "==================================================================================\n",
      "\n",
      "Evaluating router: random with threshold 0.6083487669281045...\n",
      "\n",
      "=============== random with threshold 0.6083487669281045 on gsm8k ===============\n",
      "Average accuracy: 72.303\n",
      "Model counts: gpt-4-1106-preview: 523, mistralai/Mixtral-8x7B-Instruct-v0.1: 784\n",
      "Model %: gpt-4-1106-preview: 40.015%, mistralai/Mixtral-8x7B-Instruct-v0.1: 59.985%\n",
      "=================================================================================\n",
      "\n",
      "Evaluating router: random with threshold 0.700589475694727...\n",
      "\n",
      "=============== random with threshold 0.700589475694727 on gsm8k ===============\n",
      "Average accuracy: 70.926\n",
      "Model counts: gpt-4-1106-preview: 392, mistralai/Mixtral-8x7B-Instruct-v0.1: 915\n",
      "Model %: gpt-4-1106-preview: 29.992%, mistralai/Mixtral-8x7B-Instruct-v0.1: 70.008%\n",
      "================================================================================\n",
      "\n",
      "Evaluating router: random with threshold 0.7977797711273282...\n",
      "\n",
      "=============== random with threshold 0.7977797711273282 on gsm8k ===============\n",
      "Average accuracy: 68.248\n",
      "Model counts: gpt-4-1106-preview: 262, mistralai/Mixtral-8x7B-Instruct-v0.1: 1045\n",
      "Model %: gpt-4-1106-preview: 20.046%, mistralai/Mixtral-8x7B-Instruct-v0.1: 79.954%\n",
      "=================================================================================\n",
      "\n",
      "Evaluating router: random with threshold 0.9028303733879693...\n",
      "\n",
      "=============== random with threshold 0.9028303733879693 on gsm8k ===============\n",
      "Average accuracy: 65.264\n",
      "Model counts: mistralai/Mixtral-8x7B-Instruct-v0.1: 1176, gpt-4-1106-preview: 131\n",
      "Model %: mistralai/Mixtral-8x7B-Instruct-v0.1: 89.977%, gpt-4-1106-preview: 10.023%\n",
      "=================================================================================\n",
      "\n",
      "Evaluating router: random with threshold 0.9998815592294843...\n",
      "\n",
      "=============== random with threshold 0.9998815592294843 on gsm8k ===============\n",
      "Average accuracy: 63.734\n",
      "Model counts: mistralai/Mixtral-8x7B-Instruct-v0.1: 1307\n",
      "Model %: mistralai/Mixtral-8x7B-Instruct-v0.1: 100.000%\n",
      "=================================================================================\n",
      "\n",
      "Loading model checkpoint from routellm/causal_llm_augmented ...\n",
      "Loading checkpoint shards: 100%|██████████████████| 4/4 [00:02<00:00,  1.77it/s]\n",
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n",
      "Done loading model in 7.867060899734497 seconds.\n",
      "  0%|                                                  | 0/1307 [00:00<?, ?it/s]/home/ray/anaconda3/lib/python3.11/site-packages/transformers/generation/configuration_utils.py:540: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.\n",
      "  warnings.warn(\n",
      "/home/ray/anaconda3/lib/python3.11/site-packages/transformers/generation/configuration_utils.py:545: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.\n",
      "  warnings.warn(\n",
      "The attention mask is not set and cannot be inferred from input because pad token is same as eos token.As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
      "100%|███████████████████████████████████████| 1307/1307 [06:31<00:00,  3.34it/s]\n",
      "Evaluating router: causal_llm with threshold 0.14828628301620483...\n",
      "\n",
      "=============== causal_llm with threshold 0.14828628301620483 on gsm8k ===============\n",
      "Average accuracy: 85.769\n",
      "Model counts: gpt-4-1106-preview: 1307\n",
      "Model %: gpt-4-1106-preview: 100.000%\n",
      "======================================================================================\n",
      "\n",
      "Evaluating router: causal_llm with threshold 0.2228175401687622...\n",
      "\n",
      "=============== causal_llm with threshold 0.2228175401687622 on gsm8k ===============\n",
      "Average accuracy: 84.851\n",
      "Model counts: mistralai/Mixtral-8x7B-Instruct-v0.1: 131, gpt-4-1106-preview: 1176\n",
      "Model %: mistralai/Mixtral-8x7B-Instruct-v0.1: 10.023%, gpt-4-1106-preview: 89.977%\n",
      "=====================================================================================\n",
      "\n",
      "Evaluating router: causal_llm with threshold 0.24563899040222167...\n",
      "\n",
      "=============== causal_llm with threshold 0.24563899040222167 on gsm8k ===============\n",
      "Average accuracy: 84.009\n",
      "Model counts: mistralai/Mixtral-8x7B-Instruct-v0.1: 262, gpt-4-1106-preview: 1045\n",
      "Model %: mistralai/Mixtral-8x7B-Instruct-v0.1: 20.046%, gpt-4-1106-preview: 79.954%\n",
      "======================================================================================\n",
      "\n",
      "Evaluating router: causal_llm with threshold 0.26810282468795776...\n",
      "\n",
      "=============== causal_llm with threshold 0.26810282468795776 on gsm8k ===============\n",
      "Average accuracy: 82.938\n",
      "Model counts: mistralai/Mixtral-8x7B-Instruct-v0.1: 391, gpt-4-1106-preview: 916\n",
      "Model %: mistralai/Mixtral-8x7B-Instruct-v0.1: 29.916%, gpt-4-1106-preview: 70.084%\n",
      "======================================================================================\n",
      "\n",
      "Evaluating router: causal_llm with threshold 0.28559954166412355...\n",
      "\n",
      "=============== causal_llm with threshold 0.28559954166412355 on gsm8k ===============\n",
      "Average accuracy: 80.872\n",
      "Model counts: mistralai/Mixtral-8x7B-Instruct-v0.1: 523, gpt-4-1106-preview: 784\n",
      "Model %: mistralai/Mixtral-8x7B-Instruct-v0.1: 40.015%, gpt-4-1106-preview: 59.985%\n",
      "======================================================================================\n",
      "\n",
      "Evaluating router: causal_llm with threshold 0.3040478825569153...\n",
      "\n",
      "=============== causal_llm with threshold 0.3040478825569153 on gsm8k ===============\n",
      "Average accuracy: 79.495\n",
      "Model counts: mistralai/Mixtral-8x7B-Instruct-v0.1: 653, gpt-4-1106-preview: 654\n",
      "Model %: mistralai/Mixtral-8x7B-Instruct-v0.1: 49.962%, gpt-4-1106-preview: 50.038%\n",
      "=====================================================================================\n",
      "\n",
      "Evaluating router: causal_llm with threshold 0.3240053415298462...\n",
      "\n",
      "=============== causal_llm with threshold 0.3240053415298462 on gsm8k ===============\n",
      "Average accuracy: 76.435\n",
      "Model counts: mistralai/Mixtral-8x7B-Instruct-v0.1: 784, gpt-4-1106-preview: 523\n",
      "Model %: mistralai/Mixtral-8x7B-Instruct-v0.1: 59.985%, gpt-4-1106-preview: 40.015%\n",
      "=====================================================================================\n",
      "\n",
      "Evaluating router: causal_llm with threshold 0.34344909191131595...\n",
      "\n",
      "=============== causal_llm with threshold 0.34344909191131595 on gsm8k ===============\n",
      "Average accuracy: 73.604\n",
      "Model counts: mistralai/Mixtral-8x7B-Instruct-v0.1: 915, gpt-4-1106-preview: 392\n",
      "Model %: mistralai/Mixtral-8x7B-Instruct-v0.1: 70.008%, gpt-4-1106-preview: 29.992%\n",
      "======================================================================================\n",
      "\n",
      "Evaluating router: causal_llm with threshold 0.37578885555267333...\n",
      "\n",
      "=============== causal_llm with threshold 0.37578885555267333 on gsm8k ===============\n",
      "Average accuracy: 70.926\n",
      "Model counts: mistralai/Mixtral-8x7B-Instruct-v0.1: 1045, gpt-4-1106-preview: 262\n",
      "Model %: mistralai/Mixtral-8x7B-Instruct-v0.1: 79.954%, gpt-4-1106-preview: 20.046%\n",
      "======================================================================================\n",
      "\n",
      "Evaluating router: causal_llm with threshold 0.4104311227798462...\n",
      "\n",
      "=============== causal_llm with threshold 0.4104311227798462 on gsm8k ===============\n",
      "Average accuracy: 67.559\n",
      "Model counts: mistralai/Mixtral-8x7B-Instruct-v0.1: 1176, gpt-4-1106-preview: 131\n",
      "Model %: mistralai/Mixtral-8x7B-Instruct-v0.1: 89.977%, gpt-4-1106-preview: 10.023%\n",
      "=====================================================================================\n",
      "\n",
      "Evaluating router: causal_llm with threshold 0.6229617297649384...\n",
      "\n",
      "=============== causal_llm with threshold 0.6229617297649384 on gsm8k ===============\n",
      "Average accuracy: 63.734\n",
      "Model counts: mistralai/Mixtral-8x7B-Instruct-v0.1: 1307\n",
      "Model %: mistralai/Mixtral-8x7B-Instruct-v0.1: 100.000%\n",
      "=====================================================================================\n",
      "\n",
      "mistralai/Mixtral-8x7B-Instruct-v0.1 63.733741392501905\n",
      "gpt-4-1106-preview 85.76893649579189\n",
      "Saving plot to ./gsm8k.png\n",
      "Metrics:\n",
      "        method 20% qual 50% qual 80% qual        AUC      APGR\n",
      "1  causal_llm   11.75%   34.06%   62.38%  77.540277  0.626567\n",
      "0      random   19.69%   53.05%   83.02%  74.436777  0.485725\n"
     ]
    }
   ],
   "source": [
    "!python -m routellm.evals.evaluate --config config.example.yaml --routers random causal_llm --benchmark gsm8k"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhMAAAHWCAYAAADNbgu+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAAClf0lEQVR4nOzdd3xN9x/H8dfN3gmyhEiITew9albM1t7EriqqNUrV3vqjVEtL7VFVo1WjiqK1996EGCFC9s695/fHrUskSCLJyfg8H488eu+533vO597Gve98z/d8vxpFURSEEEIIIdLISO0ChBBCCJG9SZgQQgghxDuRMCGEEEKIdyJhQgghhBDvRMKEEEIIId6JhAkhhBBCvBMJE0IIIYR4JxImhBBCCPFOJEwIIYQQ4p1ImBBCAHDixAlq1aqFtbU1Go2Gs2fPql1Strdhwwby5s1LRESE2qW8M41Gw+DBg9/Y5vLly5iYmHDx4sVMqkpkFRImRLaxYsUKNBqN4cfExIQCBQrQq1cvHjx4kOHHnz59Or/99luGH+dlL79eIyMj3NzcaNKkCfv370/X48THx9OhQweePXvGN998w+rVq/Hw8EjXY+Q2Wq2WCRMmMGTIEGxsbNQu57Vu3LhB586dKViwIFZWVpQsWZLJkycTFRWV6n2VLl2aFi1aMH78+AyoVGRlJmoXIERqTZ48mcKFCxMTE8PRo0dZsWIFBw8e5OLFi1hYWGTYcadPn0779u1p3bp1hh0jOe+//z49e/ZEURT8/PxYuHAhDRs2ZPv27TRr1ixdjnHr1i3u3r3LkiVL6NevX7rsM7f7448/uHbtGgMGDFC7lNe6d+8e1apVw97ensGDB5M3b16OHDnChAkTOHXqFL///nuq9zlw4ECaN2/OrVu38PLyyoCqRVYkYUJkO82aNaNKlSoA9OvXD0dHR2bNmsXWrVvp2LGjytWlTkxMDGZmZhgZvb6TsHjx4nTv3t1wv02bNpQrV4558+a9c5iIjIzE2tqawMBAABwcHN5pf8ntO7davnw5tWvXpkCBAmqX8lqrV68mJCSEgwcPUqZMGQAGDBiATqdj1apVBAcHkydPnlTts3HjxuTJk4eVK1cyefLkjChbZEFymkNke3Xr1gX0f12/7O+//6Zu3bpYW1vj4ODAhx9+yJUrVxK16dWrF56enkn2OXHiRDQajeG+RqMhMjKSlStXGk479OrVy/D4gwcP6NOnDy4uLpibm1OmTBmWLVuWaJ/79+9Ho9Gwfv16vvrqKwoUKICVlRVhYWGper3e3t44Ojri5+dn2Hb16lXat29P3rx5sbCwoEqVKmzdujXR856fJjpw4ACDBg3C2dmZggUL0qtXL+rVqwdAhw4d0Gg01K9fP1Xv4/P36/Lly3Tt2pU8efJQp04dADw9PWnZsiX79++nSpUqWFpa4u3tbThVs3nzZry9vbGwsKBy5cqcOXMm0b7Pnz9Pr169KFKkCBYWFri6utKnTx+ePn2abA03b96kV69eODg4YG9vT+/evZPtsl+zZg3VqlXDysqKPHny8N577/HXX38larNz507Da7e1taVFixZcunTprf+PYmJi+PPPP2ncuHGSx6Kjoxk6dCiOjo7Y2trywQcf8ODBAzQaDRMnTjS0Cw8PZ9iwYXh6emJubo6zszPvv/8+p0+fNrSpX78+ZcuW5fz589SrVw8rKyuKFi3Kxo0bAThw4ADVq1fH0tKSEiVKsGfPnkS1PP/dc3FxSbQ9f/78GBkZYWZm9sbXOXXqVIyMjFiwYIFhm6mpKfXr109Tr4bIvqRnQmR7d+7cAUj0F9SePXto1qwZRYoUYeLEiURHR7NgwQJq167N6dOnkw0Qb7J69Wr69etHtWrVDN3Wz7twHz9+TI0aNQwD1JycnNi5cyd9+/YlLCyMYcOGJdrXlClTMDMzY8SIEcTGxr71A/tVwcHBBAcHU7RoUQAuXbpk+At49OjRWFtbs2HDBlq3bs2mTZto06ZNoucPGjQIJycnxo8fT2RkJO+99x4FChRg+vTpDB06lKpVqxq+XFL7Pnbo0IFixYoxffp0FEUxbL958yZdu3blo48+onv37vzvf/+jVatW/PDDD3z55ZcMGjQIgBkzZtCxY0euXbtm6K3ZvXs3t2/fpnfv3ri6unLp0iUWL17MpUuXOHr0aKLQB9CxY0cKFy7MjBkzOH36ND/99BPOzs7MmjXL0GbSpElMnDiRWrVqMXnyZMzMzDh27Bh///03TZo0Mfw/9/X1xcfHh1mzZhEVFcWiRYuoU6cOZ86ceePv0KlTp4iLi6NSpUpJHuvVqxcbNmygR48e1KhRgwMHDtCiRYsk7QYOHMjGjRsZPHgwpUuX5unTpxw8eJArV64k2m9wcDAtW7akc+fOdOjQgUWLFtG5c2fWrl3LsGHDGDhwIF27duXrr7+mffv23Lt3D1tbW0AfRmbNmkXfvn2ZNGkS+fLl4/DhwyxatIihQ4e+sWfpq6++Yvr06fz444/0798/0WOVK1fm999/JywsDDs7u9fuQ+QgihDZxPLlyxVA2bNnj/LkyRPl3r17ysaNGxUnJyfF3NxcuXfvnqFthQoVFGdnZ+Xp06eGbefOnVOMjIyUnj17Grb5+voqHh4eSY41YcIE5dV/HtbW1oqvr2+Stn379lXy58+vBAUFJdreuXNnxd7eXomKilIURVH27dunAEqRIkUM294GUPr27as8efJECQwMVI4dO6Y0atRIAZQ5c+YoiqIojRo1Ury9vZWYmBjD83Q6nVKrVi2lWLFihm3P3786deooCQkJiY7zvLZff/010faUvo/P368uXbokeQ0eHh4KoBw+fNiwbdeuXQqgWFpaKnfv3jVs//HHHxVA2bdvn2Fbcu/Vzz//rADKP//8k6SGPn36JGrbpk0bJV++fIb7N27cUIyMjJQ2bdooWq02UVudTqcoiqKEh4crDg4OSv/+/RM9/ujRI8Xe3j7J9lf99NNPCqBcuHAh0fZTp04pgDJs2LBE23v16qUAyoQJEwzb7O3tlU8++eSNx6lXr54CKOvWrTNsu3r1qgIoRkZGytGjRw3bn7/ny5cvT7SPKVOmKJaWlgpg+Bk7dmySYwGGeoYPH64YGRkpK1asSLaudevWKYBy7NixN9Yvcg45zSGyncaNG+Pk5IS7uzvt27fH2tqarVu3UrBgQQACAgI4e/YsvXr1Im/evIbnlStXjvfff58dO3akWy2KorBp0yZatWqFoigEBQUZfnx8fAgNDU3ULQ3g6+uLpaVlio+xdOlSnJyccHZ2pnr16hw6dIjPP/+cYcOG8ezZM/7++286duxIeHi44dhPnz7Fx8eHGzduJLnSpX///hgbG7/1uGl5HwcOHJjsvkqXLk3NmjUN96tXrw5Aw4YNKVSoUJLtt2/fNmx7+b2KiYkhKCiIGjVqACR5b5OroW7dujx9+tTQpf/bb7+h0+kYP358krEqz3s5du/eTUhICF26dEn0/9TY2Jjq1auzb9++ZF/nc89Pwbw63uDPP/8EMPTEPDdkyJAk+3BwcODYsWM8fPjwjceysbGhc+fOhvslSpTAwcGBUqVKGd5PSP69Bf1pqPfee4/FixezadMm+vTpw/Tp0/nuu++SHEtRFAYPHsz8+fNZs2YNvr6+ydb0/HUHBQW9sXaRc8hpDpHtfP/99xQvXpzQ0FCWLVvGP//8g7m5ueHxu3fvAvoP1VeVKlWKXbt2pdvgwCdPnhASEsLixYtZvHhxsm2eD258rnDhwqk6xocffsjgwYPRaDTY2tpSpkwZQ+03b95EURTGjRvHuHHjXnv8lwcBpvT4aXkfX7fvlwMDgL29PQDu7u7Jbg8ODjZse/bsGZMmTWL9+vVJ3svQ0NC3Huv5F1twcDB2dnbcunULIyMjSpcunWytoL9cEvRhJzkp7bpXXjrVA/r31MjIKMn79PyU1ctmz56Nr68v7u7uVK5cmebNm9OzZ0+KFCmSqF3BggWTnOqxt7dP0Xu7fv16BgwYwPXr1w1hvG3btuh0Or744gu6dOlCvnz5DO1XrVpFREQEixYtokuXLm993a/WJXIuCRMi26lWrZrhao7WrVtTp04dunbtyrVr11J9Pf/rPuy0Wm2Knq/T6QDo3r37a/9KK1euXKL7qemVAP2XRXID+V4+/ogRI/Dx8Um2zatfVKk9fmq8bt+v6wl53faXv4Q7duzI4cOHGTlyJBUqVMDGxgadTkfTpk0Nrz+1+3yb5/tdvXo1rq6uSR43MXnzR+fzL+Dg4GDDl3RqdezYkbp167Jlyxb++usvvv76a2bNmsXmzZsTXcXzLu/twoULqVixYpIaP/jgA1asWMGZM2cS/e7Vrl2bs2fP8t1339GxY8dEPVYvex5YHB0dU/ZiRbYnYUJka8bGxsyYMYMGDRrw3XffMXr0aMNkS9euXUvS/urVqzg6Ohr+ms6TJw8hISFJ2j3/q/xlyQUPJycnbG1t0Wq1r/3Cz0jP/0o1NTVN9+On5n3MKMHBwezdu5dJkyYlmgjpec9BWnh5eaHT6bh8+TIVKlR4bRsAZ2fnNL2vJUuWBMDPzw9vb2/Ddg8PD3Q6HX5+fhQrVsyw/ebNm8nuJ3/+/AwaNIhBgwYRGBhIpUqVmDZtWrrNL/L48eNkL/2Mj48HICEhIdH2okWLMnv2bOrXr0/Tpk3Zu3evYTDny/z8/DAyMqJ48eLpUqfI+mTMhMj26tevT7Vq1Zg3bx4xMTHkz5+fChUqsHLlykRB4eLFi/z11180b97csM3Ly4vQ0FDOnz9v2BYQEMCWLVuSHMfa2jpJ8DA2NqZdu3Zs2rQp2SmEnzx58u4v8A2cnZ2pX78+P/74IwEBAel6/NS8jxnl+V/Xr/YqzJs3L837bN26NUZGRkyePDlJz8bz4/j4+GBnZ8f06dMNX6wve9v7WrlyZczMzDh58mSi7c97jxYuXJho+8uXVoK+Z+zVUzjOzs64ubkRGxv7xmOnRvHixTlz5gzXr19PtP3nn3/GyMgoSa8a6HvaduzYwZUrV2jVqhXR0dFJ2pw6dYoyZcoYTq2InE96JkSOMHLkSDp06MCKFSsYOHAgX3/9Nc2aNaNmzZr07dvXcEmjvb19omv5O3fuzBdffEGbNm0YOnSo4fK/4sWLJxncV7lyZfbs2cPcuXNxc3OjcOHCVK9enZkzZ7Jv3z6qV69O//79KV26NM+ePeP06dPs2bOHZ8+eZehr//7776lTpw7e3t7079+fIkWK8PjxY44cOcL9+/c5d+5cmved0vcxo9jZ2fHee+8xe/Zs4uPjKVCgAH/99VeiOTZSq2jRoowdO5YpU6ZQt25d2rZti7m5OSdOnMDNzY0ZM2ZgZ2fHokWL6NGjB5UqVaJz5844OTnh7+/P9u3bqV27drIDFJ+zsLCgSZMm7NmzJ9HETZUrV6Zdu3bMmzePp0+fGi4Nff5l/rz3Kzw8nIIFC9K+fXvKly+PjY0Ne/bs4cSJE8yZMyfNr/1VI0eONMylMXjwYPLly8e2bdvYuXMn/fr1w83NLdnn1ahRg99//53mzZvTvn17fvvtN0xNTQF9r8bzuUxELqLSVSRCpNrzSxtPnDiR5DGtVqt4eXkpXl5ehsse9+zZo9SuXVuxtLRU7OzslFatWimXL19O8ty//vpLKVu2rGJmZqaUKFFCWbNmTbKXhl69elV57733DJfRvXyZ6OPHj5VPPvlEcXd3V0xNTRVXV1elUaNGyuLFiw1tXnf55Zvw0uV4b3Lr1i2lZ8+eiqurq2JqaqoUKFBAadmypbJx40ZDmze9f2+qLSXv4/P368mTJ0me7+HhobRo0SJFr83Pz08BlK+//tqw7f79+0qbNm0UBwcHxd7eXunQoYPy8OHDJJdSvq6G56/bz88v0fZly5YpFStWVMzNzZU8efIo9erVU3bv3p3kffHx8VHs7e0VCwsLxcvLS+nVq5dy8uTJJK/nVZs3b1Y0Go3i7++faHtkZKTyySefKHnz5lVsbGyU1q1bK9euXVMAZebMmYqiKEpsbKwycuRIpXz58oqtra1ibW2tlC9fXlm4cGGifdWrV08pU6ZMkmOn5j0/duyY0qxZM8PvTvHixZVp06Yp8fHxb33u77//rpiYmCidOnUyXGa7c+dOBVBu3Ljx1vdI5BwaRUnFqCQhhBApotVqKV26NB07dmTKlClvbHv27FkqVqzImjVr6NatWyZVmDFat26NRqNJ9lShyLlkzIQQQmQAY2NjJk+ezPfff59oCfLkxhjMmzcPIyMj3nvvvcwsMd1duXKFbdu2vTU8iZxHeiaEECITTZo0iVOnTtGgQQNMTEzYuXMnO3fuZMCAAfz4449qlydEmkiYEEKITLR7924mTZrE5cuXiYiIoFChQvTo0YOxY8e+df4KIbIqCRNCCCGEeCcyZkIIIYQQ70TChBBCCCHeSY4/QafT6Xj48CG2tray6IwQQgiRCoqiEB4ejpubW5JVdl+W48PEw4cPk6yeJ4QQQoiUu3fv3hsXrcvxYeL5IjT37t1L8bLBQgghhICwsDDc3d2TXdDtZTk+TDw/tWFnZydhQgghhEiDtw0TkAGYQgghhHgnEiaEEEII8U4kTAghhBDinUiYEEIIIcQ7kTAhhBBCiHciYUIIIYQQ70TChBBCCCHeiYQJIYQQQrwTCRNCCCGEeCcSJoQQQgjxTiRMCCGEEOKdSJgQQgghcoDH929x8dAfPL5/K9OPneMX+jKIjARj46TbjY3BwiJxu9cxMgJLy7S1jYoCRUm+rUYDVlZpaxsdDTrd6+uwtk5b25gY0GrTp62Vlb5ugNhYSEhIn7aWlvr3GSAuDuLj06ethcWL35XUtI2P17d/HXNzMDFJfduEBP178TpmZmBqmvq2Wq3+/93rmJrq26e2rU6n/11Lj7YmJvr3AvT/JqKi0qdtav7dy2dE8m3lMyL1bTPwM+LkL3OoeHE6LhoFraLhZNkvqdJ6sP7xd/mMeNPv8MuUHC40NFQBlFD9P72kP82bJ36ClVXy7UBR6tVL3NbR8fVtq1RJ3NbD4/VtS5dO3LZ06de39fBI3LZKlde3dXRM3LZevde3tbJK3LZ589e3ffXXpn37N7eNiHjR1tf3zW0DA1+0HTTozW39/F60HTHizW0vXnzRdsKEN7c9fvxF29mz39x2374Xbb/77s1tt2170Xb58je33bDhRdsNG97cdvnyF223bXtz2+++e9F23743t509+0Xb48ff3HbChBdtL158c9sRI1609fN7c9tBg160DQx8c1tf3xdtIyLe3LZ9eyWRN7WVzwj9j3xGvPjJYp8RIfOmvLntO3xGhIICKKGhocqb5J6eCSGEECKniAmD40vg4mbsDx5Quxo0iqIoaheRkcLCwrC3tyf04UPs7OySNpAuzOTbShdm6tvKaQ79bTnNkba28hmhvy2fEa9vGx0C13fCpd/g3kEw+u93QKegxL94WwASFCOe9v4XlwJF3ukzIiwwEHs3N0JDQ5P/Dv1P7gkTb3kjhBBCiCwnNgKu/wkXN8HNPaB9ETBinMuzKKgCv0RVoaX1ZUYn/IiJRkeCYsTpchOo1m7YOx8+pd+hcppDCCGEyEriY+Dmbn2AuPYnJLzUe+dcBsq25aZzEzr9+pinUXGUdLXl436deBrSn6C7V3H0KEm1gl6ZWrKECSGEEEJt2ni4fQAuboSr2yE27MVjeYtA2fZQti04l+Lig1C6Lz1GSFQ8ZQvYsbpPdfJYm4GNFy6ZHCKekzAhhBBCqEGnhbuH9T0Ql3+H6GcvHrMrCGXbQNl2kL+CYUDE2Xsh9Fx6jLCYBMq7O7CqTzXsLU3Vqf8lEiaEEEKIzKIocP+kPkBc2gIRj148Zu0EZf4LEAWrvRgQ+p9Td5/hu+wEEbEJVPbIw4reVbG1UD9IgIQJIYQQImMpCjy+qA8QFzdBiP+LxywcoPQH+gDhUQeMk/9aPnb7Kb1XnCAqTkv1wnlZ1qsq1uZZ5ys861QihBBC5CRBN14EiKDrL7ab2UCJ5voA4dUQTMzeuJtDN4Pou/IEMfE66hR1ZEnPKliaJTOjs4pUXZtDq9Uybtw4ChcujKWlJV5eXkyZMoWXr1bt1asXGo0m0U/Tpk1VrFoIIYR4jeC7cPAb+KEOfFcF9s/QBwljcyj1AXRYCSNuQLslUKLpW4PEgetP6LNCHyTqFXfiJ9+sFyRA5Z6JWbNmsWjRIlauXEmZMmU4efIkvXv3xt7enqFDhxraNW3alOXLlxvumz+fmEYIIYRQW1gAXP5N3wNx/8SL7UYm4NVI3wNRohlYpG6uo71XHvPxmtPEaXU0LuXM990qYW6S9YIEqBwmDh8+zIcffkiLFi0A8PT05Oeff+b48eOJ2pmbm+Pq6pqifcbGxhL70uxeYWFhb2gthBBCpEHUM/0VGBc3wZ2D6JewANBA4br6AFHqA7DKm6bd/3nxEUN+Pk28VqFZWVfmd66ImUnWXehb1TBRq1YtFi9ezPXr1ylevDjnzp3j4MGDzJ07N1G7/fv34+zsTJ48eWjYsCFTp04lX758ye5zxowZTJo0KTPKF0IIkdOFPoBntyCvF5jb6ueAuLgJbu8D3UtTebtX1weI0h+Cbcr++H2dP849ZNgvZ9HqFFqVd+ObjuUxMc66QQJUnk5bp9Px5ZdfMnv2bIyNjdFqtUybNo0xY8YY2qxfvx4rKysKFy7MrVu3+PLLL7GxseHIkSMYJ7OkeHI9E+7u7jKdthBCiNQ5vQr++BSU/9Yr0ZiA8lKAyF9eHyDKtAGHQulyyC1n7jN8wzl0CrStWICvO5TH2Ejz9idmkGwxnfaGDRtYu3Yt69ato0yZMpw9e5Zhw4bh5uaGr68vAJ07dza09/b2ply5cnh5ebF//34aNWqUZJ/m5uYypkIIIUTaRT6Fcz/DX2MTb1cSIE8RqNAFyrQFx6LpetgNJ+/xxabzKAp0quLO9LbeqgaJ1FA1TIwcOZLRo0cbAoO3tzd3795lxowZhjDxqiJFiuDo6MjNmzeTDRNCCCFEqoXc05/CuLoN7h560Rvxqg/mQ+H30v3wa4/dZeyWiwB0r1GIyR+UxSibBAlQOUxERUVh9MoMX8bGxujesATu/fv3efr0Kfnz58/o8oQQQuRUigJPrsKVbXD1Dwg4l/hxp5Lw5BovBlYCGmP92Il0tuKQHxP/uAxA79qejG9ZGo0m+wQJUDlMtGrVimnTplGoUCHKlCnDmTNnmDt3Ln369AEgIiKCSZMm0a5dO1xdXbl16xajRo2iaNGi+Pj4qFm6EEKI7Eangwcn4cof+l6IZ7dePKYxgkI1oWQL/U8ez//GTAwDRasPEq3mgX2BdC1pyT+3mbbjCgAfvVeE0c1KZrsgASoPwAwPD2fcuHFs2bKFwMBA3Nzc6NKlC+PHj8fMzIzo6Ghat27NmTNnCAkJwc3NjSZNmjBlyhRcXFxSdIyUDh4RQgiRAyXEwZ1//juFsSPxWhjGZlCkAZRqCcWbgY1T0ueHPoBnt/Urd6ZzkPh+302+3nUNgMENijK8SfEsFyRS+h2qapjIDBImhBAil4mNgJt79OMfrv8FsaEvHjO3g2JN9AGiaGP95Z6ZTFEU5u+9wbw9NwD4/P3iDG1ULNPrSIlscTWHEEIIkS4ig+DaTn2AuLUPtC+mCMDGRb8WRsmW+sGTb5nCOiMpisL//rrG9/v0p1hGNS3BoPrpe1WIGiRMCCGEyJ6C7764AsP/SOIrMPIU1vc+lGwFBasmWc5bDYqiMH3HFZb86wfAVy1K0a9uEZWrSh8SJoQQQmQPigKBl/+7AmMbPDqf+HHXclCqlb4HwrkUZKHxB4qiMOmPy6w4fAeAyR+WoWdNT1VrSk8SJoQQQmRdOh3cP64PD1e2QbDfi8c0RlColr4HokRzyOOhXp1voNMpfPX7RdYd80ejgWmtvelaPX1mzMwqJEwIIYTIWhLiwO8f/fwPV3dAZOCLx4zNwauh/vLNEs3A2lG9OlNAq1MYvek8v566j0YDs9uVo0MVd7XLSncSJoQQQmS+lxfQsi8AseFwY7e+B+LGboh9acVnc3so3kR/+qJoYzC3Ua/uVEjQ6hjx6zl+O/sQIw3M7ViB1hXT9/LSrELChBBCiMyVaAEtjX58w9OboI170cbG5b8JpFqCZ11Vr8BIi3itjmG/nGX7+QCMjTR827kiLcrl3JmbJUwIIYTIPM/uwNahvJim+r9BlaDvpSjVUh8gClTJEldgpEVcgo4hP59m16XHmBpr+K5rJXzKvNuy5FmdhAkhhBAZL8QfTq2AE0tJtN7Fcx9+DxW6ZakrMNIiJl7LJ2tPs/dqIGbGRvzQoxINS6ZsxubsTMKEEEKIjKHT6sc/nFwGN/4i2RAB+nUvijTIEUFiwOpT/HP9CeYmRizpWYX3iiczRXcOJGFCCCFE+gp/DGdWwamVEHrvxfbC9aBqX4h6BtuHZ+gCWpktKi6BfitPcvjWUyxNjVnqW4VaRbP2lSbpScKEEEKId6cocOdf/WmMq9tAl6DfbuEAFbtD5d7g+NK00cWaZNgCWpktIjaBPstPcPzOM6zNjFneuxrVCudVu6xMJWFCCCFE2kU9g3M/609lPL35YnvBavpeiNIfgqll0ufZF8j2IQIgLCaeXsuOc9o/BFtzE1b0qUZljzxql5XpJEwIIYRIHUWBB6f0vRCXNkNCjH67mQ2U6whV+oCrt7o1ZoLQqHh6LjvGufuh2FmYsLpvdcq7O6hdliokTAghhEiZ2Ai4sEHfC/HowovtLt5QtQ94d1BlSW81PIuMo8fSY1x6GEYeK1NW961O2QL2apelGgkTQggh3uzRRX2AOL8B4sL124zNoWxbqNIXClbJ9ldipEZQRCzdfzrG1UfhONqYsaZfdUq62qldlqokTAghhEgqPgYu/6YPEfeOvdier6j+NEb5LmCVuwYZAgSGxdD1p2PcDIzAydacn/tXp6hz7uiNeRMJE0IIIV54eksfIM6uhehg/TYjE/2slFX6QOH3clUvxMsCQqPpuuQYfkGRuNpZsK5/dYo4ZY91QjKahAkhhMjttPFwbYc+RNze/2K7vTtU9oWKPcA2Z08H/Tb3g6PouuQY/s+iKOBgyc/9a1Aon5XaZWUZEiaEECK3Cr2vn1jq9CqIePTfRg0Ue18/FqLY+2BkrGqJagsIjebY7WfM3HmVR2ExFMprxbr+1SmYR4LEyyRMCCFEbqLTwa29+l6I63/+t3InYO0ElXpCJV/I46FujVnELyf8GbP5Arr/ZgF3tDHjl49qkN8+mXkzcjkJE0IIkRtEPIEzq+HUcv2iW8951tWPhSjZMtst852RAkKjEwUJ0F8OKpInYUIIIXKS0Afw7JZ+OW87N7h7SN8LcXkr6OL1bSzs9St0Vu4NTsXVrTeL+ud6UKIgAaBT4E5QlPRMJEPChBBC5BSnV8Efn/536kIDNs4Q8fjF4wWq6HshyrQBMznn/zqXHoYyffvlJNuNNRo8HeV9S46ECSGEyAlCH7wUJAAUfZAwsYTynfQhIn95VUvMDs7fD6HH0uOExiRQMI8lD0Oi0Sn6IDG9bVnplXgNCRNCCJETPDj1UpB4SYcVUKJpppeTHZ26G0yvZccJj02gUiEHVvSpRmRsAneCovB0tJIg8QYSJoQQIrvzPwbbRyTdrjHOFQtupYfjfs/ovfw4kXFaqnnmZVnvqtiYm2BnYSohIgWM1C5ACCFEGikKHFkIK5pD5GOwcQHNfx/rGmNoNS9HLPOd0Q7fCsJ3mT5I1PLKx4o++iAhUk7eLSGEyI5iwmDrYLj8u/5+mbbwwbf67c9uQ94iEiRS4J/rT+i/6iSxCTrqFnNkSc8qWJjm7om60kLChBBCZDePL8GGnvD0JhiZgs90qNZfv2aGua2EiBT6++pjBq4+TZxWR8OSzizsVkmCRBpJmBBCiOzk3Hr4YxgkRINdQf0AS/eqaleV7ey69IjB604Tr1XwKePCgi6VMDORM/9pJWFCCCGyg/gY+HO0fgZLAK+G0PYnsM6nbl3Z0PbzAXy6/gwJOoUW5fIzr1MFTI0lSLwLCRNCCJHVBd+BDb4QcBbQQP3R8N7IXL8IV1r8fvYBn/1yFp0CrSu48b8O5TGRIPHOJEwIIURWdu1P2DIAYkLBMi+0WwJFG6tdVba08dR9Rm48h6JA+8oFmdWuHMZGGrXLyhEkTAghRFakTYB90+DgXP39AlWg40qwL6huXdnUz8f9+XLLBRQFulYvxNQPy2IkQSLdSJgQQoisJiIQNvaBO//q71f7CJpMlVU902jVkTuM//0SAL1qeTKhVWk0GgkS6UnChBBCZCV3j8CvvSDiEZha6+eO8G6vdlXZ1k//3mbq9isA9K9bmC+bl5IgkQEkTAghRFagKHDkO9g9ARQtOJaATqvBqYTalWVbi/bfYtafVwEYVN+LkT4lJEhkEAkTQgihtphQ+P0TuPKH/r53B2g5D8xtVC0rO/t27w3m7r4OwLDGxfi0UTEJEhlIwoQQQqjp0UXY0EM/BbaRKTSdAVX76WezFKmmKApzd19nwd83ARjpU4JPGhRVuaqcT8KEEEKo5cxa2P45JMSAvTt0WAkFK6tdVbalKAozd17lx39uAzC2eSn6v1dE5apyBwkTQgiR2eKjYecoOL1Kf79oY2i7BKzyqltXNqYoCpO3XWb5oTsATGxVml61C6tbVC4iYUIIITLTMz/9Il2PzgMaaPAl1B0BRjILY1rpdArjt15kzVF/AKa1KUu36h4qV5W7SJgQQojMcnUHbBkIsaFglQ/aLQWvBmpXla3pdApjNl/gl5P30GhgVttydKzqrnZZuY6ECSGEyGjaBPh7Chyap79fsJp+tU9ZKvydaHUKIzeeY/PpBxhpYE7H8rSpKDOEqkHChBBCZKTwx/rZLO8e1N+v/jG8P1lms3xHCVodn204xx/nHmJspGFepwq0Ku+mdlm5loQJIYTIKHcOwcbeEPEYzGzgw++gTBu1q8r24hJ0fLr+DDsvPsLUWMOCLhVpWja/2mXlahImhBAivSkKHP4W9kzSz2bpVEo/m6VjMbUry/ZiE7R8svYMe648xszYiEXdK9GolIvaZeV6EiaEECI9RYfoZ7O8uk1/v1wnaPkNmFmrWlZOEBOvZeCaU+y/9gRzEyMW96xCveJOapclkDAhhBDpJ+C8/rLPYD8wNoOmM6FKH5nNMh1Ex2npv+okB28GYWFqxFLfqtQu6qh2WeI/EiaEECI9nF4F20eANhbsC0HHlVCgktpV5QiRsQn0XXmCo7efYWVmzPJeValeJJ/aZYmXSJgQQoh3ER+tDxFn1+jvF2sCbX6U2SzTSXhMPL2Xn+Dk3WBszU1Y0acqlT3kvc1qJEwIIURaPb0FG3zh8QXQGOlns6wzXGazTCeh0fH4LjvO2Xsh2FmYsKpvdSq4O6hdlkiGhAkhhEiLK9vgt48hNgysHKH9UihSX+2qcoyQqDi6Lz3GxQdhOFiZsqZvdcoWsFe7LPEaEiaEECI1tAmwdyIcXqC/714DOiwHO5kwKb08jYil20/HuPoonHzWZqzpV51S+e3ULku8gYQJIYRIidAHcP8EHPoWHp7Sb6s5GBpPBGNTVUvLSQLDY+i25Bg3AiNwsjVnXb/qFHOxVbss8RYSJoQQ4m1Or4I/huonowIwNod2S6D0h+rWlcM8Co2h609Huf0kElc7C9b1r04RJxu1yxIpIGFCCCHe5P5J2DoUUF5s08VDgSqqlZQTPQiJpuuSo9x9GkUBB0vW9a+ORz6Z6Cu7kDAhhBDJiQiEf+fCiSUkChIAig6e3ZZVP9PJvWdRdFlylPvB0bjntWRdvxq457VSuyyRChImhBDiZdHB+nERx36A+Kjk22iMIW+RzK0rh7oTFEnXJUd5GBpDYUdr1varjpuDpdpliVSSMCGEEACxEXBsERxaALGh+m1uFaHhOAi9D9s+0y/apTGGVvOkV+IdBYRGc+hGEDP/vEpQRBxeTtas618DFzsLtUsTaSBhQgiRu8XHwMml+lMaUUH6bc6locFYKNnixboaRRvrT23kLSJB4h39csKfMZsvoPvv7JGLnTnrB9TEydZc3cJEmkmYEELkTtp4OLMGDsyG8If6bXmLQP0voWxbMDJO3N6+gISIdBAQGs3oTRcSjUJ5Eh5Lgk6nWk3i3UmYEELkLjotXNgI+2foV/cEsCsA9b6ACl1lzogMFBWXwNgtF14dzopOgTtBUeS3l7ES2ZWECSFE7qAocOUP2DcdnlzRb7N2grrDoXJvMJVz9Rnp8sMwhvx8mltPIpM8ZqzR4OkoV29kZxImhBA5m6LAzb3w9xQIOKvfZmEPtT+Fah+BuUyKlJEURWH5oTvM3HmVOK0OZ1tzPqzgxrKDd9AqCsYaDdPblpVeiWxOwoQQIue6exj2TgH/w/r7ptZQ42OoNQQsHVQtLTcIiohl5K/n2HftCQCNSzkzu3158lqb0adOYe4EReHpaCVBIgeQMCGEyHkenIa/p8Ktvfr7xuZQtR/U+QxsnNStLZf45/oTPt9wjqCIWMxMjPiqRSl61PBA89/VMfntLSVE5CBGah5cq9Uybtw4ChcujKWlJV5eXkyZMgVFeTE8R1EUxo8fT/78+bG0tKRx48bcuHFDxaqFEFlW4BVY3w2WNNAHCSMT/XiIoWeg6XQJEpkgLkHH9B1X6LnsOEERsRR3sWHr4Nr0rOlpCBIi51G1Z2LWrFksWrSIlStXUqZMGU6ePEnv3r2xt7dn6NChAMyePZtvv/2WlStXUrhwYcaNG4ePjw+XL1/GwkIGTAkh0M//sH8mnN+AfuprDZTrCPVHy0yVmej2kwiGrj/DxQdhAHSvUYivWpTGwtT4Lc8U2Z1GebkbIJO1bNkSFxcXli5datjWrl07LC0tWbNmDYqi4ObmxvDhwxkxYgQAoaGhuLi4sGLFCjp37vzWY4SFhWFvb09oaCh2dnYZ9lqEECoIfQD/zNbPF6FL0G8r1Uo/4ZRzKXVry0UURWHjqftM2HqJqDgtDlamzGpXDp8yrmqXJt5RSr9DVe2ZqFWrFosXL+b69esUL16cc+fOcfDgQebOnQuAn58fjx49onHjxobn2NvbU716dY4cOZJsmIiNjSU2NtZwPywsLONfiBAic0U8gYPfwImfQPvfv/eijaHhV/opsEWmCY2O56vfLvLHOf3EXzWK5OWbThVkPEQuo2qYGD16NGFhYZQsWRJjY2O0Wi3Tpk2jW7duADx69AgAFxeXRM9zcXExPPaqGTNmMGnSpIwtXAihjugQOLwAji6C+P/mKyhUCxqNA49aqpaWG526+4yhP5/lQUg0xkYaPn+/OAPreWFsJGMjchtVw8SGDRtYu3Yt69ato0yZMpw9e5Zhw4bh5uaGr69vmvY5ZswYPv/8c8P9sLAw3N3d06tkIYQaYiP0q3ge/hZi/luEK38FfYjwavRi/QyRKbQ6he/33WT+3htodQrueS2Z37kilQrlUbs0oRJVw8TIkSMZPXq04XSFt7c3d+/eZcaMGfj6+uLqqj/f9vjxY/Lnz2943uPHj6lQoUKy+zQ3N8fcXBaLESJHiI+BU8vh3zkQqZ+rAKdS0HAslGwpIUIFD0Oi+eyXsxzzewbAhxXcmNK6LHYWMg15bqZqmIiKisLIKPHVqcbGxuj+W/ClcOHCuLq6snfvXkN4CAsL49ixY3z88ceZXa4QIrNo4+HsWv0iXGEP9NvyeOoHVpZtl3QRLpEp/rwYwBebLhAaHY+1mTFTWpelbaWCapclsgBVw0SrVq2YNm0ahQoVokyZMpw5c4a5c+fSp08fADQaDcOGDWPq1KkUK1bMcGmom5sbrVu3VrN0IURG0Ong4ibYP11/uSeArRvUGwUVu8siXCqJjtMyedtlfj7uD0D5gvbM71wRT0drlSsTWYWqYWLBggWMGzeOQYMGERgYiJubGx999BHjx483tBk1ahSRkZEMGDCAkJAQ6tSpw59//ilzTAiRU4Q+gKc3IfiOflxE4GX9ditH/SJcVfrIIlwquvwwjKHrz3AzMAKNBj56z4vP3y+OmYmqcx6KLEbVeSYyg8wzIUQWdnoVbB0KLy9KbW4PtYdA9Y9lES4VKYrCisN3mLHzKnEJ+gW6vulUgdpFHdUuTWSibDHPhBAiFwt9kDRIoIE+f4JLabWqEsDTiFhGbjzP31cDAWhU0pnZ7cuRz0YGt4vkSZgQQqjj+GISBwn096OeqlGN+M+/N/QLdD0J1y/QNbZ5KXrW9JB1NcQbSZgQQmS+2/v1k0+9SmMsa2moJC5Bx5y/rvHjP/qBr8Wcbfi2S0VK5ZfTw+LtJEwIITLX48vwSw9QtOBWCQLO6W9rjKHVPLAvoHaFuY5fUCSfrj/D+fv6CcG6Vdcv0GVpJpfgipSRMCGEyDxhAbC2A8SG6afB7vkbRAbpLwPNW0SCRCZTFIVNpx8w/veLRMVpsbfUL9DVtKws0CVSR8KEECJzxIbDuo4Qdh/yFYPOa8HEXB8gJERkurCYeL7acpGt/y3QVb1wXuZ1lgW6RNpImBBCZDxtAvzaCx6d188f0e1XsMqrdlW51mn/YIb+fIb7wfoFuj5rXIyP6xeVBbpEmkmYEEJkLEWBHcPh5h4wsYSuGyBvYbWrypW0OoVF+2/yzR79Al0F8+gX6KrsIQt0iXcjYUIIkbEOfgOnVgAaaL8UClZWu6JcKSBUv0DX0dv6Bbo+KO/G1DayQJdIHxImhBAZ58JG2DtJf7vZLCjZQt16cqk/Lz7ii03nCY2Ox8rMmMkflqVdpQIyd4RINxImhBAZ484h+O2/1X1rfALVP1K3nlwoOk7L1O2XWXtMv0CXdwF7vu1SkcKyQJdIZxImhBDp78l1WN8VtHFQqhU0map2RblGQGg0fkGRaHUKk/+4zI3ACAA+eq8Iw5uUkAW6RIaQMCGESF8RgbC2PcSEQMGq0HYJGMkXWGb45YQ/YzZfQPfSLOVOtubM7VieusWc1CtM5HgSJoQQ6ScuEtZ1gpC7kKcwdFkPpjJvQWYICI1OEiQAVvauSmk3e3WKErmG/LkghEgfOi1s6gcPT4NlHui2EaxluerMcuhGUJIgARAanZD5xYhcR8KEEOLdKQr8OQau7QBjc32PhGNRtavKNf669IjxWy8l2W6s0eDpaKVCRSK3kTAhhHh3RxfC8R/1t9v+CIVqqFtPLqHTKXyz+zoDVp8iKk5LYUdrnk9iaazRML1tWZkeW2QKGTMhhHg3l3+HXWP1t9+fAmXaqFtPLhEeE89nv5xjz5XHAPSq5cnYFqUIiojlTlAUno5WEiREppEwIYRIu3vHYfMAQIGq/aDWELUryhVuPYlgwKqT3HoSiZmJEdNal6VDFXcA8ttbSogQmU7ChBAibZ7egp87Q0IMFG8KTWeBzKiY4fZeecyw9WcJj03A1c6CH3tUpry7g9pliVxOwoQQIvUin+rnkoh6CvkrQPtlYCwfJxlJp1P4ft9N5u65jqJAVc88LOxWGSdbc7VLE0LChBAileKjYX0XeHYb7AvpVwE1k+mZM1JEbAIjNpzjz0uPAOhRw4NxLUvLbJYiy5AwIYRIOZ0OtgyEe8fA3B66/Qq2LmpXlaPdCYqk/6qT3AiMwMzYiMkflqFztUJqlyVEIhImhBApt2c8XP4NjEyh81pwLql2RTna/muBDP35DGExCTjbmvNDj8pUKpRH7bKESELChBAiZY4vgcML9LdbL4TCddWtJwdTFIVFB27x9a5rKApUKuTAD90r42xnoXZpQiRLwoQQ4u2u7YSdo/S3G34F5TqqW08OFhWXwMiN59l+PgCALtXcmfhBGcxNjFWuTIjXkzAhhHizB6dhYx9QdFCpJ9QdoXZFOZb/0ygGrD7J1UfhmBprmPhBGbpV91C7LCHeSsKEEOL1gu/qVwGNjwKvRtBirswlkUH+vfGEwevOEBodj6ONOT90r0QVz7xqlyVEikiYEEIkLzoY1naAyEBwKQsdVoCxqdpV5TiKorDk39vM3HkVnQLl3R34sXtlXO1lfITIPiRMCCGSSoiF9d0h6BrYuunnkrCwU7uqHCc6TssXm86z9dxDADpWKcjkD8tiYSrjI0T2ImFCCJGYosDvg+HuQTCz1c8lYV9A7apynPvBUQxYdYrLAWGYGGkY36o0PWp4oJHTSCIbkjAhhEjs76lwYQNojKHjSnAtq3ZFOc7hW0EMXneGZ5Fx5LM2Y2G3SlQvkk/tsoRIMwkTQogXTq2Ef/+nv91qPhRtpG49OYyiKCw7dIfpO66g1Sl4F7Dnxx6VcXOQVT5F9iZhQgihd3MPbPtMf/u9UVCph7r15DAx8Vq+3HyBzWceANC2UgGmt/GW8REiR5AwIYSAgPOwwRcULZTrDA2+VLuiHOVhSDQfrT7FhQehGBtpGNu8FL1re8r4CJFjSJgQIrcLvQ/rOkJcBHjWhQ8WyFwS6ejY7acMWnuap5Fx5LEy5ftulajl5ah2WUKkqzSvX5uQkMCePXv48ccfCQ8PB+Dhw4dERESkW3FCiAwWEwprO0J4ADiVhE5rwMRM7apyBEVRWHn4Dt1+OsbTyDhK57dj6+A6EiREjpSmnom7d+/StGlT/P39iY2N5f3338fW1pZZs2YRGxvLDz/8kN51CiHSmzZef2oj8BLYuOgvAbV0ULuqHCEmXsv43y+y4eR9AD6s4MbMtuWwNJPxESJnSlPPxKeffkqVKlUIDg7G0vLFKOQ2bdqwd+/edCtOCJFBFAX+GAa394GpFXT9BRwKqV1VjvAoNIZOi4+y4eR9jDQwtnkp5nWqIEFC5Ghp6pn4999/OXz4MGZmibtDPT09efDgQboUJoTIQP98DWfXgMYI2i8Ht4pqV5QjnLzzjIFrThMUEYuDlSkLulSkbjEntcsSIsOlKUzodDq0Wm2S7ffv38fW1vadixJCZKBz62HfNP3t5v+DEk3VrSeHWHvsLhO3XiJeq1DS1ZbFPapQKJ+V2mUJkSnSdJqjSZMmzJs3z3Bfo9EQERHBhAkTaN68eXrVJoRIb7cP6KfKBqj9KVTtq249OUBsgpYxmy8wdstF4rUKLcrlZ/OgWhIkRK6iURRFSe2T7t+/j4+PD4qicOPGDapUqcKNGzdwdHTkn3/+wdnZOSNqTZOwsDDs7e0JDQ3Fzk4WKhK5WOAVWOoDsaFQpi20WwpGab6gSwCBYTEMXHOK0/4haDQwyqckA+sVkfkjRI6R0u/QNJ3mKFiwIOfOneOXX37h3LlzRERE0LdvX7p165ZoQKYQIosIf6RfTjw2FArVhNaLJEi8o9P+wQxcfYrA8FjsLEz4tktF6pfIOn9ICZGZ0tQzkZ1Iz4TI9WIjYEVzCDgH+YpC391glVftqrK1X074M+63S8RpdRR3sWFxjyp4OlqrXZYQ6S5DeyZmzJiBi4sLffr0SbR92bJlPHnyhC+++CItuxVCpDdtAmzsrQ8SVo76uSQkSKRJQGg0Nx5H8NuZB4b1NZqWceV/HctjYy6TCYvcLU3/An788UfWrVuXZHuZMmXo3LmzhAkh1Bb6AJ7ehLNr4cZfYGIBXdZD3iJqV5Yt/XLCnzGbL6B7qR93RJPifNKgqIyPEII0holHjx6RP3/+JNudnJwICAh456KEEO/g9Cr441NQdC+2tfsJ3KuqV1M2FhAazejNF3j5hLCRBtpVLihBQoj/pGkElru7O4cOHUqy/dChQ7i5ub1zUUKINAp9kDRIoAG3SqqVlN3tufyYV0eW6RS4ExSlTkFCZEFp6pno378/w4YNIz4+noYNGwKwd+9eRo0axfDhw9O1QCFEKjy79UqQAFDg2W2wL6BKSdnZsdtPmb7jSpLtxhoNno4yj4QQz6UpTIwcOZKnT58yaNAg4uLiALCwsOCLL75gzJgx6VqgECIVooOTbtMYy1iJNDhw/QkfrT5JTLwOLydr/IIi0Sn6IDG9bVny28tl8EI8906XhkZERHDlyhUsLS0pVqwY5ubm6VlbupBLQ0Wu8fAMrGgFceEvtmmModU8qNRTtbKyo12XHjFk3RnitDoalnRmYbdKBEfFcScoCk9HKwkSItfI0EtDn7OxsaFqVRnUJYTqAq/C6rb6IOFRBz74FsIe6nsk5PRGqvx+9gGfbziHVqfQwjs/33SqgJmJEfntLSVECPEaaQoTkZGRzJw5k7179xIYGIhOl/gc7e3bt9OlOCFECgTfgdWtIfqZfqBll5/Bwg7yealdWbaz/rg/Y7bor9xoV6kgs9p5Y2IsM4UK8TZpChP9+vXjwIED9OjRg/z588vlUUKoJewhrPwAwgPAqRR036QPEiLVlh70Y8q2ywD0qOHBpA/KYGQkn21CpESawsTOnTvZvn07tWvXTu96hBApFfkUVrWGkLuQpzD0/E1mt0yj7/6+wf/+ug7AR+8VYXSzkvJHkhCpkKYwkSdPHvLmlQ8tIVQTEwpr2kLQNbB1g56/g62r2lVlO4qi8PWuayzcfwuAz98vzpCGMqulEKmVppOBU6ZMYfz48URFyaQtQmS6uChY1xkCzoJVPn2QyOOhdlXZjk6nMOmPy4Yg8VWLUgxtVEyChBBpkKaeiTlz5nDr1i1cXFzw9PTE1NQ00eOnT59Ol+KEEK9IiIMNPcD/MJjbQffN4FRc7aqyHa1OYczm82w4eR+NBqa2Lku36hLIhEirNIWJ1q1bp3MZQoi30ibA5n5wcw+YWELXDeBWQe2qsp14rY7PfjnLtvMBGGlgTsfytKlYUO2yhMjW0hQmJkyYkN51CCHeRKfTr7lx+XcwNoPOa8GjptpVZTsx8VoGrzvNniuBmBpr+LZzRZp5J120UAiROu80aZUQIhMoCuz6Es6uAY0RtFsKRRupXVW2ExWXwIBVpzh4MwhzEyN+6FGZBiWc1S5LiBwhTWFCq9XyzTffsGHDBvz9/Q3rczz37NmzdClOCAHsnwHHFulvf/g9lP5A3XqyobCYePosP8HJu8FYmRmz1LcqNb3yqV2WEDlGmq7mmDRpEnPnzqVTp06Ehoby+eef07ZtW4yMjJg4cWI6lyhELnZ4ARyYpb/d7Guo0FXderKh4Mg4uv90jJN3g7G1MGFNv+oSJIRIZ2kKE2vXrmXJkiUMHz4cExMTunTpwk8//cT48eM5evRoetcoRO50agX89ZX+dsNxUH2AquVkR4HhMXRefJTz90PJa23Gz/1rUKlQHrXLEiLHSVOYePToEd7e3oB+sa/Q0FAAWrZsyfbt29OvOiFyqwsb4Y9h+tu1P4W6w1UtJzt6GBJNpx+Pcu1xOM625mz4qAZlC9irXZYQOVKawkTBggUJCAgAwMvLi7/++guAEydOZMllyIXIVq7vgi0fAQpU6QONJ4FMpJQqd4Ii6fDDEfyCIingYMmvA2tS1NlW7bKEyLHSFCbatGnD3r17ARgyZAjjxo2jWLFi9OzZkz59+qRrgULkKn7/wC89QJcA3h2g+RwJEql043E4HX88woOQaIo4WvPrwJp45LNWuywhcjSNoijKu+7kyJEjHDlyhGLFitGqVasUP8/T05O7d+8m2T5o0CC+//576tevz4EDBxI99tFHH/HDDz+k+BhhYWHY29sTGhqKnZ2spiiysPsnYdWHEBcBJZpDx1VgbPr25wmDiw9C6bnsOM8i4yjhYsuaftVxspXeUiHSKqXfoekyz0TNmjWpWTP1E+icOHECrVZruH/x4kXef/99OnToYNjWv39/Jk+ebLhvZWX1bsUKkRU9vgRr2umDROH3oP1yCRKpdOpuML2WHyc8JoFyBe1Z2bsaeazN1C5LiFwhzWHi4cOHHDx4kMDAQHQ6XaLHhg4dmqJ9ODk5Jbo/c+ZMvLy8qFevnmGblZUVrq6yGqLIwZ7e0i8lHhMCBatC55/B1ELtqrKVw7eC6LfyJFFxWqp65mFZr6rYWkgYEyKzpClMrFixgo8++ggzMzPy5cuXaJU9jUaT4jDxsri4ONasWcPnn3+eaH9r165lzZo1uLq60qpVK8aNG/fG3onY2FhiY2MN98PCwlJdixCZJvS+PkhEBoJLWej2K5jbqF1VtrLvaiAD15wiNkFH3WKO/NijMlZmMrmvEJkpTf/ixo0bx/jx4xkzZgxGRmkaw5nEb7/9RkhICL169TJs69q1Kx4eHri5uXH+/Hm++OILrl27xubNm1+7nxkzZjBp0qR0qUmIDBXxRD9GItQf8npBjy1gKXMgpMaOCwF8uv4M8VqFxqVc+K5rRSxMjdUuS4hcJ00DMPPly8fx48fx8vJKt0J8fHwwMzPjjz/+eG2bv//+m0aNGnHz5s3XHju5ngl3d3cZgCmylugQWNkSHl0Au4LQ509wcFe7qmxl06n7jNx4Dp0Crcq7MbdjeUyN0+ePGyGEXkoHYKbpX17fvn359ddf01zcq+7evcuePXvo16/fG9tVr14dgJs3b762jbm5OXZ2dol+hMhS4iJhXUd9kLB2gp6/S5BIpTVH7zL8V32Q6FTFnXmdKkiQEEJFaTrNMWPGDFq2bMmff/6Jt7c3pqaJBzrNnTs3Vftbvnw5zs7OtGjR4o3tzp49C0D+/LJksMimEmJhfTe4dwws7KHHb+BYVO2qspUl/9xm2o4rAPSq5cn4lqUxMpK5OIRQU5rDxK5duyhRogRAkgGYqaHT6Vi+fDm+vr6YmLwo59atW6xbt47mzZuTL18+zp8/z2effcZ7771HuXLl0lK2EOrSJsDGPnB7H5haQ7dN4FpW7aqyDUVRmL/3BvP23ABgUH0vRvqUSPVnjhAi/aUpTMyZM4dly5YlGiyZVnv27MHf3z/JzJlmZmbs2bOHefPmERkZibu7O+3ateOrr75652MKkel0Ovj9E7i6DYzNoMs6cK+qdlXZhqIozNh5lcX/3AZgpE8JPmkgPTpCZBVpChPm5ubUrl07XQpo0qQJyY0BdXd3TzL7pRDZkqLAzpFwfj1ojKHDSihSX+2qsg2dTmH81ousOeoPwPiWpelTp7DKVQkhXpamEUuffvopCxYsSO9ahMiZ9k6GEz8BGmjzI5RsrnZF2UaCVseIjedYc9QfjQZmtfOWICFEFpSmnonjx4/z999/s23bNsqUKZNkAOab5oEQIlf5dy4c/G9Acsu5UK7Dm9sLg7gEHcN+OcOOC48wNtIwt2N5PqxQQO2yhBDJSFOYcHBwoG3btuldixA5y/ElsPe/CdQaT9IvJy5SJCZey8drTrHv2hPMjI34rmtFmpSRafWFyKpSHSYSEhJo0KABTZo0kTUzhHidc7/AjhH623WHQ51hqpaTnUTGJtBv5UmO3H6KhakRi3tU4b3iTm9/ohBCNakeM2FiYsLAgQMTzTIphHjJlW3w28f629UGQMNx6taTjYRGx9Nj6TGO3H6KjbkJq/pUlyAhRDaQpgGY1apV48yZM+ldixDZ3619sLE3KFoo3xWazgKZByFFnkbE0nXJUU77h2BvacqaftWpVjiv2mUJIVIgTWMmBg0axPDhw7l//z6VK1fG2to60eMyqZTIlfyPwfquoI2DUq3ggwWQTgvh5WQBodGcvhvM17uucedpFI42ZqzuW51S+WUqfCGyizQt9JXcSqEajQZFUdBoNGi12nQpLj2kdJESId5JwHlY0RJiQ8GrIXRZDybmaleV5f1ywp8xmy+g++9TyM7ChC2f1MbLSZZhFyIrSOl3aJp6Jvz8/NJcmBA5TtANWN1GHyTca0CnNRIkUiAgNJrRmy/w8p8zEbEJWJnJEuJCZDdpChMeHh7pXYcQ2VOIP6z6EKKCwLUcdNsAZtZvf14uFxmbwOQ/LvNqv6hOgTtBUeS3t1SnMCFEmqQpTIB+Ia558+Zx5Yp+9b7SpUvz6aef4uXllW7FCZGlhT/WB4mwB+BYHHps0a8EKt7o76uPGffbJR6ERCd5zFijwdPRSoWqhBDvIk2jw3bt2kXp0qU5fvw45cqVo1y5chw7dowyZcqwe/fu9K5RiKwn6hmsbg3PboN9If1S4taOaleVpQWGxfDJ2tP0WXGSByHRFMxjSe/anhj/d7WLsUbD9LZlpVdCiGwoTQMwK1asiI+PDzNnzky0ffTo0fz111+cPn063Qp8VzIAU6Sr0Afw+AL8PRUeXQAbF+i9E/JJj9zr6HQKP5/wZ+bOq4THJGBspKFvncIMa1wMKzMTAkKjuRMUhaejlQQJIbKYlH6HpilMWFhYcOHCBYoVK5Zo+/Xr1ylXrhwxMTGprziDSJgQ6eb0KvjjU1B0+vumVtBvL7iUVreuLOzG43DGbL7AybvBAJQraM+Mtt6UcZPTQUJkBxl6NYeTkxNnz55NEibOnj2Ls7NzWnYpRNYW+iBxkABIiJExEq8RE69l4b6bLDpwi3itgpWZMSOalMC3lifGRjKJlxA5TZrCRP/+/RkwYAC3b9+mVq1aABw6dIhZs2bx+eefp2uBQmQJQdcSBwnQ3392G+xlJcuXHbn1lLFbLnA7KBKARiWdmdy6LAUc5BSGEDlVmsLEuHHjsLW1Zc6cOYwZMwYANzc3Jk6cyNChQ9O1QCFUp42Ho4uSbtcYQ94imV9PFhUcGcf0HVf49dR9AJxtzZn4QRmalXVFI1OKC5GjpXjMxNatW2nWrBmmpqaJtoeHhwNga2ub/tWlAxkzId6JNgE294NLW/ThAUXfI6ExhlbzoFJPtStUnaIobD33kMl/XOZpZBwA3WsUYlTTkthZmL7l2UKIrCzdx0y0adOGR48e4eTkhLGxMQEBATg7O2fZECHEO9MmwJaP9EHCyBQ6rdZPTPXstr5HQk5v4P80irG/XeDfG0EAFHexYUZbbyp7yAJdQuQmKQ4TTk5OHD16lFatWhnW4BAix9Jp9cuIX9wIRibQcSWUaKZ/TEIE8VodSw/6MW/PdWLidZiZGDG0YVEGvOeFmYksbiZEbpPiMDFw4EA+/PBDNBoNGo0GV1fX17bNSgt9CZFqOi38/glc2KA/ndF+OZRsoXZVWca5eyGM3nyBKwFhANQsko9pbcpSRBbnEiLXSnGYmDhxIp07d+bmzZt88MEHLF++HAcHhwwsTQgV6HSwdQic+/m/ILEMSn+gdlVZQkRsAv/bdY2VR+6gKOBgZcrY5qVoX7mg9FQKkcul6mqOkiVLUqJECXx9fWnXrh02NvKXiMhBdDr4YyicXQsaI2i3BMq0VruqLGH35ceM//0iAaH6CenaVizA2BalyGcjq6MKIdIwA6ZOp8PCwoJLly4lmbQqK5KrOUSK6HSw/TM4tUIfJNouAe/2alelukehMUzceok/Lz0CoFBeK6a1KUvdYk4qVyaEyAwZNgOmkZERxYoV4+nTp9kiTAjxVooCO0a8CBJtfsz1QUKnU1h77C6z/7xGeKx+PY0B7xVhaMNiWJoZq12eECKLSdOkVTNnzmTkyJEsWrSIsmXLpndNQmQeRYEdI+HkUkADrRdBuY5qV6Wqa4/CGbP5PKf9QwCo4O7AjLbelMovPXtCiOSlKUz07NmTqKgoypcvj5mZGZaWiafJffbsWboUJ0SGUhT4czScWAJo4MPvoXxntatSTUy8lgV/3+DHA7dJ0CnYmJsw0qcE3Wt4yHoaQog3SlOYmDdvXjqXIUQmUxTYNRaO/aC//8ECqNhN3ZpUdOhmEGO3XODO0ygAmpR2YdKHZWRJcCFEiqQpTPj6+qZ3HUJkHkWB3ePg6Pf6+63mQ6Ue6takkmeRcUzbfoVNp/XrabjYmTPpg7I0Lfv6eWSEEOJVaQoTALdu3WL58uXcunWL+fPn4+zszM6dOylUqBBlypRJzxqFSD+KAnsmwuEF+vst5kLlXmpWpApFUdhy5gFTtl0mOCoejQZ61vBghE8JbGU9DSFEKqVp3tsDBw7g7e3NsWPH2Lx5MxEREQCcO3eOCRMmpGuBQqQbRYG/p8Chefr7zf8HVfuqWpIa7gRF0n3pMT7fcI7gqHhKutqy6eNaTPqwrAQJIUSapClMjB49mqlTp7J7927MzMwM2xs2bMjRo0fTrTgh0tW+6fDvHP3tZrOhWn9168lk8Vod3++7ic+8fzh08ynmJkaMalqCP4bUoVKhPGqXJ4TIxtJ0muPChQusW7cuyXZnZ2eCgoLeuSgh0t3+mfDPbP1tnxlQ/SN168lkp/2D+XLzBa4+CgegTlFHprUpi0c+a5UrE0LkBGkKEw4ODgQEBFC4cOFE28+cOUOBArKioshiDnwN+2fobzeZCjUHqVtPJggIjcYvKBInG3NWH73L6qN3URTIa23GuJalaF2hgKynIYRIN2kKE507d+aLL77g119/RaPRoNPpOHToECNGjKBnz57pXaMQaffvHNg3VX+78SSoNUTdejLBLyf8GbP5ArpXJspvX7kgXzYvRV5rs+SfKIQQaZSmMDF9+nQGDx5MoUKFSEhIoHTp0mi1Wrp27cpXX32V3jUKkTYH58HeyfrbjcZDnWFqVpMpAkKjGb35Aq+uuLOgSwValZdeQyFExkhVmNDpdHz99dds3bqVuLg4evToQbt27YiIiKBixYqyVofIOg59C3v+u7KowVdQd7i69WSC64/D+WpL0iAB4GhjkfkFCSFyjVSFiWnTpjFx4kQaN26MpaUl69atQ1EUli1bllH1CZF6R77XT0oFUH8M1Bupbj0ZzC8okvl7rvP7uYfJBgljjQZPR6vML0wIkWukKkysWrWKhQsX8tFH+pHwe/bsoUWLFvz0008YGaXpKlMh0tfRRbDrS/3t90ZB/dHq1pOBHoRE8+2eG2w8fR/tfwMkmpV1pWR+W77dcxOtomCs0TC9bVmZFlsIkaE0ipLc3zLJMzc35+bNm7i7uxu2WVhYcPPmTQoWLJghBb6rlK7FLnKAY4th53+9EHWHQ8NxkAOvWAgMi+H7fTf5+fg94rQ6ABqWdObz94tTtoA9oB87cScoCk9HKwkSQog0S+l3aKp6JhISErCwSHzu1dTUlPj4+LRVKUR6OfHTiyBRe1iODBLPIuP44cAtVh6+Q2yCPkTULpqPz98vQWWPxJNO5be3lBAhhMg0qQoTiqLQq1cvzM3NDdtiYmIYOHAg1tYvJr/ZvHlz+lUoxNucXA7b/xtgWWsoNJ6Yo4JEaHQ8P/17m2UH/YiM0wJQ2SMPw5sUp5aXo8rVCSFEKsNEcquFdu/ePd2KESLVTq2EbcP0t2sOhvcn55ggERmbwIrDd/jxwC3CYhIAKFvAjuFNSlC/uJNMOiWEyDJSFSaWL1+eUXUIkXpn1sAfn+pvV/9YP7tlDviCjYnXsuboXRbuv8WzyDgAirvY8Pn7JfAp4yIhQgiR5aR5CXIhVHV2Hfw+GFCg2kfQdEa2DxJxCTp+OeHPgr9vEhgeC4BnPis+e784Lcu5YWyUvV+fECLnkjAhsp9zv8BvgwAFqvaDZrOydZBI0OrYfPoB8/fe4EFINAAFHCz5tFEx2lYqgImxXHYthMjaJEyI7OX8r/DbQECByr2h2dfZNkjodAp/nH/IvD038AuKBMDZ1pwhDYvSsao75ibGKlcohBApI2FCZB8XN8GWAaDooFJPaDEXsuFkaYqisOvSY77ZfZ1rj/VLgue1NuPjel50r+GBpZmECCFE9iJhQmQPl7bApv76IFGxO7Scn+2ChKIoHLj+hDl/XefCg1AAbC1M+Oi9IvSqXRgbc/nnKITInuTTS2R9l7fCxr6gaKF8V2i1INsFiSO3njLnr2ucvBsMgJWZMX1qF6Z/3SLYW5mqXJ0QQrwbCRMia7uyDTb21geJcp3hw++yVZA47R/MnL+ucejmUwDMTYzoWdODgfW8yGdj/pZnCyFE9iBhQmRd13bCr71AlwDeHaD1QjDKHuMJLj4IZe7u6/x9NRAAU2MNXaoV4pMGRXGxk+XAhRA5i4QJkTVd3wW/9ABdPJRtB61/yBZB4sbjcL7Zc50dFx4BYGykoX2lggxpVJSCeWQZcCFEziRhQmQ9N3bDL931QaJ0a2izGIyz9q/qnaBI5u+9wW9nH6Ao+qtVPyjvxqeNilHEyUbt8oQQIkNl7U9okfvc3APru4E2Dkp9AO1+ytJB4kFINAv23uDXU/fR6hQAmpZx5bP3i1PC1Vbl6oQQInNk3U9pkbuEPtDPI/H3VNDGQsmW0H4ZGGetKx0CQqPxC4rE1tyEjafu8/Pxe8Rp9cuB1y/hxPD3S+Bd0F7lKoUQInNJmBDqO70K/hgKiv4ve1y8of3yLBckfjnhz5jNF/ivA8KgRpG8jGhSgiqeedUpTAghVCZhQqgr9IF+5U/lpW/owMsQ+QTsC6hX1ysCQqMZvflCojIBFnSpQKvyWadOIYRQQ/a5YF/kTA9P62e1fJmihWe31annNfZcfpwkSAA42shlnkIIIWFCqEebAIcXJN2uMYa8RTK/ntc4fz+EmX9eTbLdWKPB01Eu9xRCCAkTQj17JsC9Y2BsBpr/fhU1xtBqXpY5xXHxQSjdfzpGZKyWwvmsMfpvgVJjjYbpbcuS395S3QKFECILkDETQh0XNsKR7/S32y6BglX1pzbyFslSQaLbT8cIi0mgikceVvSpRnhMPHeCovB0tJIgIYQQ/5EwITLfo4uwdYj+du1hUKa1/nYWCREAlx+G0X3pMUKj46lUyIHlvatiY26CjbmJhAghhHiFnOYQmSs6WD+7ZXwUFGkAjcarXVESVx+F0e2no4RExVPe3YEVfapha5G1LlMVQoisRMKEyDw6HWweAMF+YF9IPylVFltv4/rjcLotOUZwVDzlCtqzqk817CRICCHEG0mYEJln/wy48ReYWEDnNWCVtSZ5uvE4nK5LjvI0Mo6yBexY3ac69pYSJIQQ4m0kTIjMcXU7/DNbf7vVfMhfXt16XnEzMIIuS44RFBFH6fx2rOlbHXsrCRJCCJESEiZExgu6AZs/0t+u9hGU76xuPa+4/SSCrkuOEhQRS6n8dqztVx0HKzO1yxJCiGxDwoTIWLHh+lVA48KhUC3wmaZ2RYn4BUXSZclRAsNjKelqy9p+1cljLUFCCCFSQ9Uw4enpiUajSfLzySefABATE8Mnn3xCvnz5sLGxoV27djx+/FjNkkVqKAr89jEEXQPb/NBhRZZavOvu00i6LD7K47BYSrjog0ReCRJCCJFqqoaJEydOEBAQYPjZvXs3AB06dADgs88+448//uDXX3/lwIEDPHz4kLZt26pZskiNg9/AlT/AyBQ6rgZbF7UrMvB/GkWXxUd5FBZDMWcb1vavTj4bc7XLEkKIbEnVSaucnJwS3Z85cyZeXl7Uq1eP0NBQli5dyrp162jYsCEAy5cvp1SpUhw9epQaNWqoUbJIqZt74e8p+tvNvwb3qurW85J7z6LosuQoD0Nj8HKyZl3/GjhKkBBCiDTLMmMm4uLiWLNmDX369EGj0XDq1Cni4+Np3LixoU3JkiUpVKgQR44cee1+YmNjCQsLS/QjMlnwHdjYR78aaMUeULmX2hUZ3A/WB4kHIdEUcbTm5/41cLKVICGEEO8iy4SJ3377jZCQEHr16gXAo0ePMDMzw8HBIVE7FxcXHj169Nr9zJgxA3t7e8OPu7t7BlYtkoiLgvXdISYE3CpB8/+BRqN2VQA8DImmy5Kj3A+OprCjNT8PqIGznSwhLoQQ7yrLhImlS5fSrFkz3Nzc3mk/Y8aMITQ01PBz7969dKpQvJWiwB+fwuMLYOUInVaDadb4sg4I1QeJe8+i8chnxc/9a+AiQUIIIdJFlljo6+7du+zZs4fNmzcbtrm6uhIXF0dISEii3onHjx/j6ur62n2Zm5tjbi7d1qo49iNc2KBfRrzjSrAvqHZFADwOi6HrkmPcfRpFobz6IOFqL0FCCCHSS5bomVi+fDnOzs60aNHCsK1y5cqYmpqyd+9ew7Zr167h7+9PzZo11ShTvMmdQ7DrS/3tJlPBs4669fwnMCyGLouP4hcUScE8lvw8oAZuDrLqpxBCpCfVeyZ0Oh3Lly/H19cXE5MX5djb29O3b18+//xz8ubNi52dHUOGDKFmzZpyJUdWE/oAfvUFRQveHaDGx2pXBEBgeAxdlhzldlAkBRws+bl/DQpIkBBCiHSnepjYs2cP/v7+9OnTJ8lj33zzDUZGRrRr147Y2Fh8fHxYuHChClWK10qIhQ09IfIJuJSFVt9miQGXT8Jj6brkGLeeROJmb8H6ATVwz2uldllCCJEjaRRFUdQuIiOFhYVhb29PaGgodnZ2apeT82wdCqdXgoUDDNgPeQurXRFBEbF0XXKU648jyP9fkPDIZ612WUIIke2k9Ds0S4yZENnUqRX6IIEG2i3NEkHiWWQc3X86xvXHEbjYmfNzfwkSQgiR0SRMiLS5fxJ2jNTfbvgVFGv85vaZIDgyjq5LjnL1UTjOtvog4ekoQUIIITKahAmRehGB8EsP0MZByZZQd7jaFRESFUe3n45x9VE4jjbmrOtfgyJONmqXJYQQuYKECZE62nj4tReEPwTH4tB6keoDLkOj4um+9BiXA8JwtDFj/YDqFHWWICGEEJlFwoRInb/Gwd1DYGYLndaChbqDWkOj4+mx7BgXH4SRz9qMdf1rUNTZVtWahBAit5EwIVLu3C9wbJH+dpsfwKm4quWExcTTc9lxzt8PJa+1GWv7V6e4iwQJIYTIbBImRMoEnNevuwFQdwSUaqlqOeEx8fguO865eyE4WJmypm91SrrKpb9CCKEGCRPi7aKewS/dICEaijaGBl+qWk5EbAK9lp/gjH8I9pamrO1XndJuEiSEEEItEibEm+m0sKkvhPhDHk9o9xMYGatWTmRsAr2XH+fU3WDsLExY2686ZdzsVatHCCGEhAnxNn9PhVt/g4mlfsClZR7VSomKS6D3ihOcuBOMrYUJa/vVoGwBCRJCCKE2CRPi9S5vhYNz9bc//A5cy6pWSnSclr4rTnLc7xm25ias6Vsd74ISJIQQIiuQMCGSF3gVfvtv9c8an4B3e9VKiYnX0m/VCY7cfoqNuQmr+lajvLuDavUIIYRITMKESComVD/gMi4CPOvC+5PVKyVeS/9VJzl08ynWZsas7FOVioXUO9UihBAiKQkTIjGdDrYMhKc3wa4AtF8OxuqsVB8Tr2XA6lP8eyMIKzNjVvSpRmWPvKrUIoQQ4vXU+ZYQWde/c+DaDjA2h06rwcZJlTJiE7QMXHOKf64/wdLUmOW9qlLVM/sHCa1WS3x8vNplCCEEAMbGxpiYmKB5x2URJEyIF27shn3T9LdbzIEClVUpIzZBy8drTrP/2hMsTI1Y1qsq1YvkU6WW9BQREcH9+/dRFEXtUoQQwsDKyor8+fNjZmaW5n1ImBB6T2/p55NAgcq9oVIPVcqIS9DxydrT/H01EHMTI5b5VqWmV/YPElqtlvv372NlZYWTk9M7/xUghBDvSlEU4uLiePLkCX5+fhQrVgwjo7SNfpAwISAuUr+keEwoFKwKzWapUka8VsfgdafZc0UfJJb6VqVWUUdVaklv8fHxKIqCk5MTlpaWapcjhBAAWFpaYmpqyt27d4mLi8PCwiJN+5EwkdspCvw+GAIvgbUzdFwNJuaZXob/s0iGbzjHiTvBmJkYsaRnFeoUyxlB4mXSIyGEyGrS2hvxMgkTud2R7+DSZjAygY6rwC5/ppew7thdvtxy0XC/W/VCvFdcnYGfQgghUk8uDc3Nbh+A3eP1t31mgEfNTC/hYUhUoiABsOrwXQJCozO9FiGEEGkjYSK3CrkHG3uDooPyXaBaf1XKmPPX9STbtIrCnaAoFaoRaurVqxetW7dWuwwhRBpImMiN4mNgQw+Iegqu5aDlN6DCufyfj/uz6fSDJNuNNRo8Ha0yvR4hhBBpI2Eit1EU2D4cHp4By7zQaQ2YZv7VBXsuP2bslgsANCzpjPF/YcZYo2F627Lkt5crHl4nIDSaw7eCVDkVFBcXl+nHFEJkfRImcpuTS+HsGtAYQftlkMcj00s47R/M4J9Po1OgQ+WCLPWtwsHRDfi5fw0Ojm5Ap6qFMr2mzKYoClFxCan+WX3kDrVn/k3XJceoPfNvVh+5k+p9pGbSrPr16zN48GCGDRuGo6MjPj4+zJ07F29vb6ytrXF3d2fQoEFEREQYnrNixQocHBzYtWsXpUqVwsbGhqZNmxIQEGBoo9Vq+fzzz3FwcCBfvnyMGjUqSV2xsbEMHToUZ2dnLCwsqFOnDidOnDA8vn//fjQaDbt27aJixYpYWlrSsGFDAgMD2blzJ6VKlcLOzo6uXbsSFSWnzYTISHI1R27ifwx2jtbfbjQBvBpkegm3nkTQd8UJYuJ11C/hxPS23mg0GvLbW+aq3ojoeC2lx+96p33oFBj3+yXG/X4pVc+7PNkHK7OU/9NfuXIlH3/8MYcOHQJg586dfPvttxQuXJjbt28zaNAgRo0axcKFCw3PiYqK4n//+x+rV6/GyMiI7t27M2LECNauXQvAnDlzWLFiBcuWLaNUqVLMmTOHLVu20LBhQ8M+Ro0axaZNm1i5ciUeHh7Mnj0bHx8fbt68Sd68L6ZWnzhxIt999x1WVlZ07NiRjh07Ym5uzrp164iIiKBNmzYsWLCAL774IlXvkxAi5SRM5Bbhj2BDT9DFQ+nWUPvTTC8hMCwG32XHCY6Kp3xBexZ2q4SpsXSOZXXFihVj9uzZhvslSpQw3Pb09GTq1KkMHDgwUZiIj4/nhx9+wMvLC4DBgwczefKL1WfnzZvHmDFjaNu2LQA//PADu3a9CFeRkZEsWrSIFStW0KxZMwCWLFnC7t27Wbp0KSNHjjS0nTp1KrVr1wagb9++jBkzhlu3blGkSBEA2rdvz759+yRMCJGBJEzkBglxsMEXIh6BUyn48PtMH3AZHhNPr+UnuB8cjWc+K5b1qpqqv45zGktTYy5P9knVcx6FxtB47gF0L50NMNLAns/r4Wqf8lnrLE2NU3XcypUTr9GyZ88eZsyYwdWrVwkLCyMhIYGYmBiioqKwstIPnLWysjIECYD8+fMTGBgIQGhoKAEBAVSvXt3wuImJCVWqVDGc6rh16xbx8fGGkABgampKtWrVuHLlSqJ6ypUrZ7jt4uKClZWVIUg833b8+PFUvWYhROrIn4U5WegD8PsH/hgK946CuR10XgvmNplaRlyCjoFrTnE5IAxHGzNW9alOPpvMn2UzK9FoNFiZmaTqp4iTDTPaeicarDqjrTdFnGxStZ/UzsJpbW1tuH3nzh1atmxJuXLl2LRpE6dOneL7778HEg/ONDU1TfJ6M2qBs5ePpdFokj22TqfLkGMLIfRy75+GOd3pVfDHp/p5JJ5ruwTyeb3+ORlAp1MYufEch24+xdrMmOW9qlEon1z2mVadqupnB70TFIWno1WmjzM5deoUOp2OOXPmGKbg3bBhQ6r2YW9vT/78+Tl27BjvvfceAAkJCZw6dYpKlSoB4OXlhZmZGYcOHcLDQz9IOD4+nhMnTjBs2LD0e0FCiHQhYSInCn2QNEigAVfvTC9l5p9X+f3sQ0yMNCzqXhnvgvaZXkNOo+Zg1aJFixIfH8+CBQto1aoVhw4d4ocffkj1fj799FNmzpxJsWLFKFmyJHPnziUkJMTwuLW1NR9//DEjR44kb968FCpUiNmzZxMVFUXfvn3T8RUJIdKDnObIiZ7deiVIACjw7HamlrH0oB+L/9Efc3b7crLeRg5Qvnx55s6dy6xZsyhbtixr165lxowZqd7P8OHD6dGjB76+vtSsWRNbW1vatGmTqM3MmTNp164dPXr0oFKlSty8eZNdu3aRJ0+e9Ho5Qoh0olEy6kRmFhEWFoa9vT2hoaHY2dmpXU7mCH0A35QBXvpfqzGGYRfAvkCmlLDt/EOG/HwGRYFRTUswqH7RTDluVhUTE4Ofnx+FCxdO8xK/QgiREd70+ZTS71DpmciJHl8iSZBoNS/TgsThW0F8/ss5FAV8a3rwcb3MHachhBAic8mYiZwmMgh+/0R/u2JPKNcR8hbJtCBxJSCMj1adIk6ro7m3K+NblUn11QNCCCGyFwkTOYmi6AdeRgaCU0loPjtT1914EBJNr+XHCY9NoFrhvMztWAFjIwkSQgiR08lpjpzk7Fq4ug2MTKHt4kwNEiFRcfguO87jsFiKu9iwpEcVLFI5OZIQQojsScJEThF8B3b+N11wgy8hf/lMO3RMvJa+K09yMzCC/PYWrOxTDXsr07c/UQghRI4gYSIn0Glh80cQFwGFambquhtancLQn89w6m4wdhYmrOxTLVct2CWEEELCRM5waL5+umwzW2jzAxhlzukFRVGYsPUif11+jJmJET/5VqW4i22mHFsIIUTWIWEiuws4B/um6283mwV5PDPt0N/vu8mao/5oNDC/UwWqFc779icJIYTIcSRMZGfx0bB5gH5Z8VKtoELXTDv0hpP3+N9f1wGY2KoMzbzzZ9qxhRBCZC0SJrKzPZPgyVWwcYGW8zNtWfF9VwMZs/kCAB/X98K3lmemHFeI5Gg0Gn777be3trtz5w4ajYazZ88CsH//fjQaTaI1QYQQaSNhIru6tQ+OLdLf/vB7sM6XKYc9ey+EQWtPo9UptK1UgFE+JTLluEIIIbIuCRPZUdQz+G2Q/naVvlDs/Uw57J2gSPqsOEF0vJb3ijsxq105md1SDaEPwO8f/X+FECILkDCRHe0YAeEPIV9RaDIlUw75JDyWnsuO8ywyDu8C9izqVglTY/n1STNFgbjI1P8cXwLzysLKVvr/Hl+S+n2kcm0/nU7H7NmzKVq0KObm5hQqVIhp06YB8MUXX1C8eHGsrKwoUqQI48aNIz4+3vDcXr160bp160T7GzZsGPXr1zfc37hxI97e3lhaWpIvXz4aN25MZGQkACdOnOD999/H0dERe3t76tWrx+nTp9P2nr/FihUrcHBwYNu2bZQoUQIrKyvat29PVFQUK1euxNPTkzx58jB06FC0Wm2G1CBEdiXTaWc353+Fi5v0i3e1WQxm1hl+yMjYBPqsOIH/syg88lmxrFdVrM3lV+edxEfBdLd324ei0wfLHSNS97wvH6bq92bMmDEsWbKEb775hjp16hAQEMDVq1cBsLW1ZcWKFbi5uXHhwgX69++Pra0to0aNStG+AwIC6NKlC7Nnz6ZNmzaEh4fz77//8nwx4/DwcHx9fVmwYAGKojBnzhyaN2/OjRs3sLVN/8uQo6Ki+Pbbb1m/fj3h4eG0bduWNm3a4ODgwI4dO7h9+zbt2rWjdu3adOrUKd2PL0R2Jd8I2Unofdg+XH+73hdQsHKGHzJeq+Pjtae58CCUfNZmrOxdDSdb8ww/rsgawsPDmT9/Pt999x2+vr4AeHl5UadOHQC++uorQ1tPT09GjBjB+vXrUxUmEhISaNu2LR4eHgB4e3sbHm/YsGGi9osXL8bBwYEDBw7QsmXLd3ptyYmPj2fRokV4eelXum3fvj2rV6/m8ePH2NjYULp0aRo0aMC+ffskTAjxEgkT2YVOB1sGQmwoFKgCdYdn+CEVReGLTef55/oTLE2NWdarKp6OGd8TkiuYWul7CFIj7CF8X03fI/Gcxhg+OQZ2qejlMLVKcdMrV64QGxtLo0aNkn38l19+4dtvv+XWrVtERESQkJCAnZ1divdfvnx5GjVqhLe3Nz4+PjRp0oT27duTJ08eAB4/fsxXX33F/v37CQwMRKvVEhUVhb+/f4qPkRpWVlaGIAHg4uKCp6cnNjY2ibYFBgZmyPGFyK7kpHd2cWwR3PlX/0XQdjEYZ3wOnL3rGptPP8DYSMPC7pUo7+6Q4cfMNTQa/amG1Pw4FoNW8/UBAvT/bTVPvz01+0nFoFlLy9dPjX7kyBG6detG8+bN2bZtG2fOnGHs2LHExcUZ2hgZGRlOWTz38pgKY2Njdu/ezc6dOyldujQLFiygRIkS+Pn5AeDr68vZs2eZP38+hw8f5uzZs+TLly/RMdKTqWniNWU0Gk2y23Q6HUKIFyRMZAePL+vnlADwmQb5vN7cPh2sPHyHRftvATCzrTcNSjhn+DFFClTqCcMugO82/X8r9czQwxUrVgxLS0v27t2b5LHDhw/j4eHB2LFjqVKlCsWKFePu3buJ2jg5OREQEJBo2/N5Hp7TaDTUrl2bSZMmcebMGczMzNiyZQsAhw4dYujQoTRv3pwyZcpgbm5OUFBQ+r5IIcQ7k9McWV1CrH6WS20sFPOByr0z/JA7LwQw8Y9LAIxoUpwOVdwz/JgiFewL6H8ygYWFBV988QWjRo3CzMyM2rVr8+TJEy5dukSxYsXw9/dn/fr1VK1ale3btxtCwHMNGzbk66+/ZtWqVdSsWZM1a9Zw8eJFKlasCMCxY8fYu3cvTZo0wdnZmWPHjvHkyRNKlSoF6MPM6tWrqVKlCmFhYYwcOfKNvSVCCHVIz0RWt28aPL4AVvnggwUZPsvlcb9nfPrLWRQFetTw4JMGRTP0eCLrGzduHMOHD2f8+PGUKlWKTp06ERgYyAcffMBnn33G4MGDqVChAocPH2bcuHGJnuvj48O4ceMYNWoUVatWJTw8nJ49X/Sm2NnZ8c8//9C8eXOKFy/OV199xZw5c2jWrBkAS5cuJTg4mEqVKtGjRw+GDh2Ks7P0kgmR1WiUV09o5jBhYWHY29sTGhqaqoFhWcKdQ7CiBaBAp7VQKv1Hr7/s+uNw2i86TFhMAj5lXFjYrTLGRjIpVXqIiYnBz8+PwoULY2FhoXY5Qghh8KbPp5R+h0rPRFYVE6q/egMFKnbP8CDxMCQa32XHCYtJoIpHHuZ3rihBQgghRIpImMiqdo6GUH9w8ICmMzP0UKFR8fRafpyA0BiKOtvwk28VLEyNM/SYQqS36dOnY2Njk+zP89MmQoiMIQMws6LLv8O5daAx0l8Gap7+M/09FxOvpf/qk1x/HIGLnTkr+1TDwcosw44nREYZOHAgHTt2TPYxGbQpRMaSMJHVhD+CPz7V367zGRSqkWGH0uoUPvvlLMf9nmFrbsLKPtUo4CAfuiJ7yps3L3nz5lW7DCFyJTnNkZUoCvz+CUQHQ/7yUG90Bh5KYfIfl9h58RFmxkYs7lmFkq7ZbICqEEKILEHCRFZy4ie4uQdMLPSLeJlk3OmGHw7cZuWRu2g0MLdTeWp65cuwYwkhhMjZJExkFUE34K//rtFvPAmcS2bYoTafvs+sP/WrPo5rUZqW5d5x9UohhBC5moSJrEAbD5v7Q0I0FGkA1QZk2KEOXH/CqI3nAfjovSL0qVM4w44lhBAid5AwkRUcmA0Pz4CFA7ReCEYZ87/lwv1QPl5zigSdQusKbnzRNON6P4QQQuQeEibUdu84/Ps//e2W36RuKelUuPs0kt4rjhMVp6VuMUdmty+PkUxKJdLZ/v370Wg0hISEZPqxV6xYgYODQ6qe4+npybx581StIT2pfXyRe0mYUFNshH4RL0UH3h2hbNsMOczTiFh8lx0nKCKOMm52LOpeGTMT+V8v0l+tWrUICAjA3t7+rW3VDB7PnThxggEDkp5WLFy4MHv27DHUmCdPHmJiYpI8V6PRoHlpvZxOnTpx/fr1FB9frfdg3rx5lChRAktLS9zd3fnss8+SvL436dWrl+G1v/xTpkyZ17bJly8fTZs25fz582/c9507d9BoNElWl02rzA5Y9evXZ9iwYemyr/3791OpUiXMzc0pWrQoK1aseGP7mJgYevXqhbe3NyYmJrRu3Tpd6kgJ+UZR064vIdgP7ApC86/TffcBodHsu/qY7j8d487TKArmsWR576rYmMv0IiJjmJmZ4erqmugL9l3FxcWl275e5eTkhJWVVaJt58+fJzg4mHr16hm22draJlkRdenSpRQqVCjRNktLywxZiCw934N169YxevRoJkyYwJUrV1i6dCm//PILX375ZYr3MX/+fAICAgw/9+7dI2/evHTo0CFRu6ZNmxra7N27FxMTE1q2zNilAdIqI3/P0sLPz48WLVrQoEEDzp49y7Bhw+jXrx+7du167XO0Wi2WlpYMHTqUxo0bZ2K1EibUc20nnF4JaKDND2DpkK67/+WEP7Vn/k3vFSe58igcKzNjVvWphrOtLDKVlcTFxb32JyEhIcVt4+PjU9Q2terXr8+QIUMYNmwYefLkwcXFhSVLlhAZGUnv3r2xtbWlaNGi7Ny5E0j6l/bdu3dp1aoVefLkwdramjJlyrBjxw7u3LlDgwYNAMiTJw8ajYZevXoZjjl48GCGDRuGo6MjPj4+AMydOxdvb2+sra1xd3dn0KBBREREvLb2W7du8eGHH+Li4oKNjQ1Vq1Zlz549idokd5rj999/p2nTppiamhq2+fr6smzZMsP96Oho1q9fj6+vb6LnvvxXsKIoNG7cGB8fH56vp/js2TMKFizI+PHjM+U9SM7hw4epXbs2Xbt2xdPTkyZNmtClSxeOHz8OwJMnT3B1dWX69OmJnmNmZsbevXsBsLe3x9XV1fBz8uRJgoOD6d27d6JjmZubG9pUqFCB0aNHc+/ePZ48eZLiep//Tu3du5cqVapgZWVFrVq1uHbtmqHNuXPnaNCgAba2ttjZ2VG5cmVOnjzJ/v376d27N6GhoYYekokTJwL6//dTpkyhZ8+e2NnZMWDAgGR7is6ePYtGo+HOnTuGbYcOHaJ+/fpYWVmRJ08efHx8CA4OplevXhw4cID58+cbjvfy855bvHgxbm5u6HS6RNs//PBD+vTpA8APP/xA4cKFmTNnDqVKlWLw4MG0b9+eb7755rXvlbW1NYsWLaJ///64urqm+D1OD/InqhoinsDWIfrbNT+BwnXTdfcBodGM3nyBl9eDjYnXYmkm621kNTNmzHjtY8WKFaNr166G+//73/+ShIbnPDw8DF9EoP/LMSoqKkm7CRMmpLrGlStXMmrUKI4fP84vv/zCxx9/zJYtW2jTpg1ffvkl33zzDT169MDf3z/Jcz/55BPi4uL4559/sLa25vLly9jY2ODu7s6mTZto164d165dw87OLtGU1ytXruTjjz/m0KFDhm1GRkZ8++23FC5cmNu3bzNo0CBGjRrFwoULk607IiKC5s2bM23aNMzNzVm1ahWtWrXi2rVrSXoUXrZ161Y+//zzRNt69OjB119/jb+/P4UKFWLTpk14enpSqVKl1+5Ho9GwcuVKvL29+fbbb/n0008ZOHAgBQoUYPz48Wg0mgx/D5JTq1Yt1qxZw/Hjx6lWrRq3b99mx44d9OjRA9D31ixbtozWrVvTpEkTSpQoQY8ePRg8eDCNGjVKdp9Lly6lcePGeHh4vPa4ERERrFmzhqJFi5IvX+rntRk7dixz5szBycmJgQMH0qdPH8N7061bNypWrMiiRYswNjbm7NmzmJqaUqtWLebNm8f48eMN4cPGxsawz//973+MHz/e8O/i3r17b63j7NmzNGrUiD59+jB//nxMTEzYt28fWq2W+fPnc/36dcqWLcvkyZMB/fv5qg4dOjBkyBD27dtneE+fPXvGn3/+yY4dOwA4cuRIkt4FHx+fdDuFkt5UDxMPHjzgiy++YOfOnURFRVG0aFGWL19OlSpVAP15t5UrVyZ6jo+PD3/++aca5b47RYE/hkLkE3AuA43Gp+vuA8NiGL3pPK8uLK9T4E5QFPntZbpskTrly5fnq6++AmDMmDHMnDkTR0dH+vfvD8D48eNZtGhRsufC/f39adeuHd7e3gAUKVLE8Njzqa+dnZ2TnNMuVqwYs2fPTrTt5Q9RT09Ppk6dysCBA1/7RVq+fHnKly9vuD9lyhS2bNnC1q1bGTx4cLLPefDgAefPn0+yMJizszPNmjVjxYoVjB8/nmXLlhn+gnyTAgUK8OOPP9KzZ08ePXrEjh07OHPmDCYmJpnyHiSna9euBAUFUadOHRRFISEhgYEDByY6zdG8eXP69+9Pt27dqFKlCtbW1q8Nvg8fPmTnzp2sW7cuyWPbtm0zfHlHRkaSP39+tm3bhlEarlibNm2a4dTT6NGjadGiBTExMVhYWODv78/IkSMpWVJ/hVqxYsUMz7O3t0ej0ST7l3rDhg0ZPny44X5KwsTs2bOpUqVKovf85bEiZmZmWFlZvbFnIE+ePDRr1ox169YZwsTGjRtxdHQ09FY9evQIFxeXRM9zcXEhLCyM6OjoLLfejKphIjg4mNq1a9OgQQN27tyJk5MTN27cIE+ePInaNW3alOXLlxvum5ubZ3ap6ef0Kri2A4zN9It4maTPa4mJ17L0oB8L990kMk6b5HFjjQZPR6tkninUNGbMmNc+9uoH7ogRI17b9tUxCp9++um7FfaScuXKGW4bGxuTL18+QzgADB94gYGB2NklnpJ96NChfPzxx/z11180btyYdu3aJdrf61SuXDnJtj179jBjxgyuXr1KWFgYCQkJxMTEEBUVlWTcA+j/Ep44cSLbt28nICCAhIQEoqOjk+1BeW7r1q3UqVMn2QF7ffr04dNPP6V79+4cOXKEX3/9lX///fetr6VDhw5s2bKFmTNnsmjRokRfdG/yru+Bv78/pUuXNtz/8ssv+fLLL9m/fz/Tp09n4cKFVK9enZs3b/Lpp58yZcoUxo0bZ2j/v//9j7Jly/Lrr79y6tSp137urly5EgcHh2QH+zVo0IBFixYB+s/7hQsX0qxZM44fP46HhwfNmjUzvIceHh5cunTpte/Hy783+fPnB/S/c4UKFeLzzz+nX79+rF69msaNG9OhQwe8vLxeu6/nnv/Rmhpnz55NMjbkbcqUKcPdu3cBqFu3Ljt37qRbt27079+fhQsXYm5uztq1a+ncuXOaglZWoGqYmDVrFu7u7omCQuHCSSdRen7eLdt7dhv+/O/Lo+E4cC37zrtUFIU/Lz5i+s4r3HsWDUAFdwdqeeXjxwO30SoKxhoN09uWlV6JLMjMLOVTpmdU27d5eewA6IPLy9ueB5lXz/8C9OvXDx8fH7Zv385ff/3FjBkzmDNnDkOGDHnjMa2trRPdv3PnDi1btuTjjz9m2rRp5M2bl4MHD9K3b1/i4uKSDRMjRoxg9+7d/O9//6No0aJYWlrSvn37N44d2bp1Kx988EGyjzVr1owBAwbQt29fWrVqleKu+qioKE6dOoWxsTE3btxI0XPg3d8DNze3RFdEPO8FGTduHD169KBfv34AeHt7ExkZyYABAxg7dqzhy+zWrVs8fPgQnU7HnTt3EgXI5xRFYdmyZfTo0SPZ3zlra2uKFi1quP/TTz9hb2/PkiVLmDp1Kj/99BPR0frPrVd/z171pt+5iRMn0rVrV7Zv387OnTuZMGEC69evp02bNm/c56vv8fPXrrzUtfvqqcW09Ajs2LHDsJ/nz2/VqhWKorB9+3aqVq3Kv//+m2g8hKurK48fP060n8ePHyc5HZZVqBomtm7dio+PDx06dODAgQMUKFCAQYMGGbpPn9u/fz/Ozs7kyZOHhg0bMnXq1Nf+Q46NjSU2NtZwPywsLENfQ4ppE2DzRxAfCR519GMl3tGlh6FM/uMyx/yeAeBqZ8EXzUrwYfkCGBlp6FHTgztBUXg6WkmQEKpxd3dn4MCBDBw4kDFjxrBkyRKGDBli+PLRapP2pL3q1KlT6HQ65syZY/jA37Bhwxufc+jQIXr16mX4QomIiEh2MNxzERER7Nu3z/CX9KtMTEzo2bMns2fPNgw4TYnhw4djZGTEzp07ad68OS1atKBhw4YAGfoemJiYJPoify4qKirJX7/GxvrxVM+/ROPi4ujevTudOnWiRIkS9OvXjwsXLiS5UuXAgQPcvHmTvn37vrV+0IcAIyMjQ4AoUKBAip6XEsWLF6d48eJ89tlndOnSheXLl9OmTRvMzMxS9P7Ci/ENAQEBhh7yVy9RLVeuHHv37mXSpEnJ7iO54yU3lsTCwoK2bduydu1abt68SYkSJRKNwalZs6Zh/MRzu3fvpmbNmil6LZlN1f6U27dvG7r9du3axccff8zQoUMTjZFo2rQpq1atYu/evcyaNYsDBw7QrFmz1/5yzJgxA3t7e8OPu7t7Zr2cNzv0Ddw/DuZ20GYRGKV9MOST8FjGbD5PywUHOeb3DHMTI4Y2KsbfI+rRpmJBw2RU+e0tqemVT4KEUM2wYcPYtWsXfn5+nD59mn379lGqVClA/wGr0WjYtm0bT548eeNVCUWLFiU+Pp4FCxZw+/ZtVq9ezQ8//PDGYxcrVozNmzdz9uxZzp07R9euXZPtPXnuzz//pHjx4nh6er62zZQpU3jy5Inh6oq32b59O8uWLWPt2rW8//77jBw5El9fX4KDg4GMfw+S06pVKxYtWsT69evx8/Nj9+7djBs3jlatWhlCxdixYwkNDeXbb7/liy++oHjx4smOEVm6dCnVq1enbNnke1ljY2N59OgRjx494sqVKwwZMoSIiAhatWqV6rpfJzo6msGDB7N//37u3r3LoUOHOHHihOH3zNPTk4iICPbu3UtQUFCyA5OfK1q0KO7u7kycOJEbN26wfft25syZk6jNmDFjOHHiBIMGDeL8+fNcvXqVRYsWERQUZDjesWPHuHPnDkFBQW/8nevWrZvhd6Rbt26JHhs4cCC3b99m1KhRXL16lYULF7JhwwY+++wzQ5vvvvsuyaDYy5cvc/bsWZ49e0ZoaChnz55Ntzk73khRkampqVKzZs1E24YMGaLUqFHjtc+5deuWAih79uxJ9vGYmBglNDTU8HPv3j0FUEJDQ9O19lS5f0pRJuVVlAl2inL25zTvJiY+Qflh/02lzPg/FY8vtikeX2xTBq87rdwPjkrHYkVGiI6OVi5fvqxER0erXUqq1KtXT/n0008TbfPw8FC++eabRNsAZcuWLcq+ffsUQAkODlYURVEGDx6seHl5Kebm5oqTk5PSo0cPJSgoyPC8yZMnK66uropGo1F8fX1fe0xFUZS5c+cq+fPnVywtLRUfHx9l1apViY61fPlyxd7e3tDez89PadCggWJpaam4u7sr3333XZJ9v/xaunfvrowdOzbRMV99Pa/asmWL8vLH6Ms1BAYGKi4uLsr06dMNj8fFxSmVK1dWOnbsmCnvQXLi4+OViRMnKl5eXoqFhYXi7u6uDBo0yLCPffv2KSYmJsq///5reI6fn59iZ2enLFy40LAtJCREsbS0VBYvXpzscXx9fRXA8GNra6tUrVpV2bhx4xvr8/PzUwDlzJkzhnpe/X9w5swZBVD8/PyU2NhYpXPnzoq7u7tiZmamuLm5KYMHD070b23gwIFKvnz5FECZMGGCoijJ/x4riqIcPHhQ8fb2ViwsLJS6desqv/76q+FYz+3fv1+pVauWYm5urjg4OCg+Pj6G+q5du6bUqFFDsbS0TPK8V2m1WiV//vwKoNy6dSvJ4/v27VMqVKigmJmZKUWKFFGWL1+e6PEJEyYoHh4eibZ5eHgket+f/7zJmz6fQkNDU/QdqlGUV8f9Zx4PDw/ef/99fvrpJ8O2RYsWMXXqVB48ePDa5zk5OTF16lQ++uijtx4jLCwMe3t7QkNDkwwOyxRxUbC4HgRdh9KtocMKSOWEPoqisPvyY6btuMLdp/pU7V3AngmtSlPFM2/61yzSXUxMDH5+fhQuXBgLC5nrI6tJSEjAxcWFnTt3Uq1aNbXLESJTvenzKaXfoaqOmahdu3aiiUcArl+//sZrle/fv8/Tp08No3mzvD0T9EHCxlW/9kYqg8TVR2FM2XaZQzefAuBka84onxK0q1RQ1tYQIp08e/aMzz77jKpVq6pdihDZkqph4rPPPqNWrVpMnz6djh07cvz4cRYvXszixYsB/YCoSZMm0a5dO1xdXbl16xajRo2iaNGiKT5nqaqbe+C4/rXQ+nuwSnkvwtOIWObuvs7Px/3RKWBmYkT/uoX5uH5RmQ5biHTm7OxsmEtDCJF6qn4rVa1alS1btjBmzBgmT55M4cKFmTdvnmEgirGxMefPn2flypWEhITg5uZGkyZNmDJlStafayLqGfz23xUb1QZA0ZTNkx6XoGPVkTvM33uD8Bj9dMrNvV0Z06wU7nllngghhBBZj+p/4rZs2fK1C79YWlq+cVGTLEtRYNswiHgEjsWhcfKXECV+isK+a4FM3XaF20GRAJTOb8f4VqWpUST1U88KIYQQmUX1MJEjnf8FLv8ORib6WS7N3tyjcONxOJO3XebfG/pLixxtzBjpU4L2ld0xlnERQgghsjgJE+ktxB92jNTfrj8a3Cq+vmlUHN/svs6aY/5odQpmxkb0ruPJ4AZFsbV482xwQgghRFYhYSI96bSw5WOIDYOC1aD2Z8k2i9fqWHv0Lt/suUFotH6K1SalXRjbohQe+ayTfY4QQgiRVUmYSE9Hvoe7B8HUGtr+CMZJ39791wKZuv0KNwP1M92VdLVlfMvS1CrqmNnVCiGEEOlCwkR6eXQR/p6iv910BuQtkujhW08imLrtMvuuPQEgr7UZw5sUp1MVd0yMs+cqcUIIIQRImEgf8TGweQBo46BEc6jU0/BQaFQ88/feYNWROyToFEyMNPSq5cmQRsWwt5RxEUKIjNWrVy9CQkL47bff1C5F5GDyJ3F6+HsKBF4Caydo9S1oNCRodaw+epf6/9vHskN+JOgUGpV05q/P3uOrlqUlSIhcRaPRpPrLbP369Wg0Glq3bv3WtosXL6Z+/frY2dmh0WgICQlJ0mbatGnUqlULKysrHBwckt2Pv78/LVq0wMrKCmdnZ0aOHElCQkKiNrGxsYwdOxYPDw/Mzc3x9PRk2bJlqXptmWn+/PmsWLFC7TJEDic9E+/K7x/9WAmADxaAjRMHbwQxZdtlrj0OB6CYsw3jWpbmveJOKhYqRPZx584dRowYQd26dVPUPioqiqZNm9K0aVPGjBmTbJu4uDg6dOhAzZo1Wbp0aZLHtVotLVq0wNXVlcOHDxMQEEDPnj0xNTVl+vTphnYdO3bk8ePHLF26lKJFixIQEPDGlSHTKi4uzrBE+buwt7dPh2qEeIs3LgOWA6R0xbM0iQpWlDml9auB/j5E8XsSofRdccKwomf5SbuUlYf9lPgEbfofW2Qrr12VLyLi9T+paRsVlbK2aRAWFqZ07dpVsbKyUlxdXZW5c+cmWtXSw8NDmTx5stK5c2fFyur/7d15VFNn+gfwbyBhX2UVZa2ouBaxIDIFTsW6TcXRY61SS91at1HU2roMtXUO6rjg0nbcasFWrNUZQcvosVSFAlVQCFpABQWXolEBWQQhkDy/Pyj355XFIFHUPp9zco553ydvnjzE5M29773XiBwcHOjLL78UHv/oVQwfvcrho+rr62nw4MH09ddfU2hoKAUHB2uc6+Ou8knU8pU1jxw5Qjo6OqRQKIS2rVu3kpmZGdXW1hIR0dGjR8nc3JxKSko0zomo4Sqgc+bMoTlz5pCZmRlZWVnRP/7xD1Kr1UJMYx0nT55MpqamwhVEk5OT6S9/+QsZGBhQ165d6e9//zvd/+NvuXTpUvL29m7yfP369aPPP/+ciKhJDVUqFa1atYpcXFzIwMCA+vXrRwcOHBD6vby8aN26dcL94OBgkkqlVFlZSUQkXIk5Pz+/TTVgzy9tXDWUd3O0x9GPgYrfobZwxTpJKIZuTMLPF25D9491EYkfBeI9XxdeYMlaZmLS8m3cOHGsrW3LsSNGiGNdXJqPewILFy5EamoqDh8+jISEBCQnJyMzM1MUs27dOvTv3x9yuRxLlizB/PnzkZCQAAA4c+YMACAqKgq3bt0S7rdk5cqVsLW1xbRp054o3yd16tQp9O3bF3Z2dkLbsGHDUFFRgZycHADA4cOHMXDgQKxduxZdunRB9+7d8dFHH+HBgwePHX/37t2QSqVIT0/H5s2bERkZKbpiMgCsX79eqGN4eDiuXLmC4cOHY9y4cTh//jx++OEHpKSkYO7cuQCAkJAQpKen48qVK8IYOTk5OH/+PCZNmtRsHqtXr8a3336Lbdu2IScnBwsWLMC7776LpKQkAEBAQAASExMBNJyZNzk5GRYWFkhJSQEAJCUloUuXLujWrZuGlWV/Bryb4wnc/v0KatO+gdNvP0ANHUytmI7EVAUAIKC7DcL/6oFutqYdnCVj7VdZWYndu3dj7969GDJkCICGSYGDg4Mozs/PD0uWLAEAdO/eHampqdi4cSOGDh0KG5uG3XsWFhawt7dv9flSUlKwa9cuZGVlaf/FPIZCoRBNJAAI9xWKhv/fBQUFSElJgYGBAWJjY1FcXIzZs2ejpKQEUVFRrY7v6OiIjRs3QiKRoEePHvjtt9+wceNGzJgxQ4h54403sGjRIuH+9OnTERISgrCwMACAu7s7tmzZgoCAAGzduhW9e/dG//79sXfvXoSHhwMAYmJi4OPj0+yXfW1tLVatWoWff/4Zvr6+AAA3NzekpKRg+/btCAgIQGBgIHbt2gWVSoXs7Gzo6elhwoQJSExMxPDhw5GYmIiAgIA2Vpe97P40kwmlUgmlUtmkXUdHB1KpVBTXEolEAvnhr+B1/jPoSghKSHFS1R8pNU7oZm2AJcN7Cusi6urqIJP9/yLLuro6EFGL4z6NWACifa5tia2vr291P3BbYmUyGSR/XHr9acWqVCqoVCqtxEqlUujo6Gg1VqlUimpPRA33KyqajZdIJJD88b4UYv/4Qms2VldXHFtQ0HwskVAHIbYFEokEBQUFqKurw2uvvSb8LUxNTdGjRw8QkdA2aNAg0WsbNGgQNm/eLPr7qdVqqNVqSCQS3LhxA7169RL6li5dirlz52Ly5MnYvn07OnXq1ORvT0RYtWoVVq9eLbRlZ2fDyclJyPfh2JbeO821q9VqoR6P5vxonEQiwXfffSesRVi/fj3efvttfPnllzh79ixGjRolxG/dulW4cKGPj4+o5j4+PtiwYQNUKhV0//j7eXl5iZ7z3LlzOH/+PGJiYpq8titXrsDDwwOTJk1CVFQUli9fDiLC999/j4ULF4riGx+Tl5eH6upqDB06VPS6lEolPD09oVar4efnh8rKSmRkZODUqVPw9/eHv78/1q5dC6Bhy8TixYsf+/5p/H/xcA4tkUgkbXpfcqx2Yxvf/0qlEjKZTHg/qtXqVr8TH/anmUxs2LABBgYGTdrd3d1FmwPXr1+Purq6ZsfobG+LaTcbJhIAsBnTUS01wmSpHKgCUv+bgtQ/Yh0cHES/OL766iuUl5c3O66NjQ1mz54t3N+5cyfu3r3bbKy5ubnwKwUAoqOjcfPmzWZjjYyMsHjxYuF+TEwMrl271mysTCbDsmXLhPv79+9Hfn5+s7EAsGLFCuHfsbGxyM3NbTF26dKlwuQjPj4e586dazH2o48+grFxw1lAjx07hrNnz7YYO3/+fGFV/vHjx3Hq1KkWY2fNmgVbW1sAQHJysrBJtznTp09Hly5dAACnT5/Gzz//3GJsaGgoXFxcAAAZGRk4evRos3HGxsZ48803hfsPHjxo9oiDRpaWljD84/1aU1ODe/futRhrYWEBI0NDAA2/PEtLS1uMNdfREeqrVCpRUlLSYqyZmZnw77q6OuHXeeP9qqoqKBQKqFQq0QdOfX09KioqoFKpRI8pKyuDQqGAsbExHBwckJWVhfr6epSUlMDCwgJpaWm4evUqgoODhcc0frFKpVLk5uZizJgxCAwMFPp1dHSE5zD8owZAwwemooXJV3l5eZMPU4VCARMTExQVFYked/36dQAQtqh07twZ9vb2ePDggbBrw8bGBkQEuVwONze3JltVFAoFlEolHjx4IBq7srJSFNc4GX20Zu+++y4++OADWFn9/wX/SkpKYGxsDIVCgTfeeANLlixBQkICampqcOPGDUyYMEGIra2tRW1tLRQKhfD/f/fu3bC3t4eOjo6w5UhfXx+lpaVQKpXo1asX4uPjkZGRgddffx09evSAXC5HXl4e8vPzERAQgNLSUtTW1jZbYwCirVf37t1DTU1Ni7H29vbCl11ZWVmru43s7OyEL7vy8nJUV1e3GGtrayv8WKyoqEBVVVWLsTY2NsIPtfv37zf5+zzM2tpa+EyrqqpCRQs/CgDAyspKuMp1dXV1i98DANCpUyfhe0qjz4g/3vMafUYYNVwjqrnPiPr6epSXl+PIkSPw9/eHt7c3gIb3//bt21sc92F/msmENtTVVAsTCcb+DNzc3CCTyYQvFaDhQ7mgoAA+Pj5C3KPrIDIzM+Hu7i7cl8lkoq02UqkU3bp1Q319vTBpMTQ0xPHjx0XjbNiwATU1Ndi8eTMcHR1x7949WFpaav11Ag1bBbZs2YLi4mJYWzeckfaXX36BqampsBXFz88P+/fvR1VVlTApKygogI6ODjp37gxDQ0M4OjoKYyoUCmFCJJfLRc+XkZEBd3d34YuxOX379kVeXh5cXV2FyTDQMNFrPGTVwcEBgwYNwsGDB1FTUwN/f39R7MO6d+8OfX19FBUVwdfXFzo6OqJdT8XFDRcbHDRoEFJTU5GVlYVPPvkElpaW6NatGyIiItC5c2d079691Yko+xPSZKXni6xxJerdu3eptra2ya2urk4U31xM4+33wotU/6l5w9EbK8yodkUnqvrUmm4UXGwSq1QqReMqlcoWx31asY0r0J8ktq6uTmuxD69Yf1qx9fX1WotVqVRajy0vL6ecnBxhtbRarSaVStXi7eF8Ozp2+vTp5OrqSj///DOdP3+exo4dS6ampjR//nxSqVTk7OxMZmZm9K9//YsuXbpEX3zxBenq6tKRI0eEsdzd3WnmzJlUVFQkOhLicTk8fCRCa7FFRUWUmZlJO3fuJACUlJREGRkZdPfuXSGmsLCQMjIy6LPPPiMTExOSy+Ukl8upsrKSVCoVKZVK6tOnDw0dOpQyMzPpyJEjZGNjQ0uWLBHyrayspK5du9K4cePot99+o5MnT5K7uztNmzZNeJ6HNbYFBASQiYkJhYWFUW5uLu3Zs4eMjY1p27ZtQqyzszNFRkaKXpdcLidDQ0OaPXs2yeVyysvLo7i4OJo9e7Yobvv27eTg4EDW1ta0e/duUQ7vvfcejR49WohdtmwZWVlZ0TfffEN5eXmUkZFBW7ZsoejoaCHm4MGDpKurS/b29kLbvHnzSFdXl9555x2N/nYP6+j3MMe2HltVVUU5OTlUXl5O9fX1ovfv3bt3NTqa408zmdDWoaFp/9lIdZ9aEK0wo7pPLSjtPxu1Mi57ubV26NXzrrlDQ729vYUvWWdnZ/r8889p/PjxQszmzZtFYxw+fJi6detGUqn0sYeGPkzTQ0NXrFghOvy08RYVFSUaq7mYkydPCjFXr16lESNGkKGhIVlbW9OiRYua/OC4cOECBQUFkaGhIXXt2pUWLlxI1Y8emvuIgIAAmj17Ns2cOZPMzMzI0tKSli1b1uTQ0I0bNzZ5bHp6Og0dOpRMTEzI2NiY+vXrRxEREaKYe/fukb6+PhkZGQmHcD78uh+uoVqtpk2bNlGPHj1IJpORjY0NDRs2jJKSkoSYkpISkkgkNGHCBKEtNjaWAIgmQOzloI1DQyVErazaeAlUVFTA3Nwc5eXlon3A7XH79ysovnYR1s49Ydf1Fa2MyV5uNTU1KCwshKura7Nrd14kVVVV6NKlCzZs2IBp06bBxcUFYWFhorU8TCwwMBCvvvoqNm3a1NGpMNZEa59Pmn6H8pqJJ2DX9RWeRLA/DblcjosXL8Lb2xvl5eVYuXIlAIgWSjLG/tx4MsEYe6z169fj0qVL0NPTg5eXF5KTk4VFiowxxpMJxlirPD09kZGR0WL/1atXn10yL6jGM0oy9rLi8zwzxhhjrF14MsHYM/SSr3dmjL2AtPG5xJMJxp6BxhMTaXpqWsYYe1YazyL68KUa2orXTDD2DEilUhgZGeHu3buQyWSi6xYwxlhHICJUV1fjzp07sLCwaPVsrI/DkwnGngGJRILOnTujsLCwxeujMMZYR9Dkir6Pw5MJxp4RPT09uLu7864Oxthz4+GrhLYHTyYYe4Z0dHRe+DNgMsbYo3jHLWOMMcbahScTjDHGGGsXnkwwxhhjrF1e+jUTjSfjqKio6OBMGGOMsRdL43fn405s9dJPJiorKwEAjo6OHZwJY4wx9mKqrKyEubl5i/0SesnP76tWq3Hz5k2YmppCIpFoZcyKigo4Ojrixo0brV7fnWmOa6pdXE/t45pqF9dT+55GTYkIlZWVcHBwaPVkey/9lgkdHR107dr1qYxtZmbG/wm0jGuqXVxP7eOaahfXU/u0XdPWtkg04gWYjDHGGGsXnkwwxhhjrF14MvEE9PX1sWLFCujr63d0Ki8Nrql2cT21j2uqXVxP7evImr70CzAZY4wx9nTxlgnGGGOMtQtPJhhjjDHWLjyZYIwxxli78GSCMcYYY+3Ck4kn8NVXX8HFxQUGBgbw8fFBenp6R6f0Qli9ejVee+01mJqawtbWFmPGjMGlS5dEMTU1NZgzZw6srKxgYmKCcePG4fbt2x2U8YtlzZo1kEgkCAsLE9q4nm1XVFSEd999F1ZWVjA0NETfvn1x9uxZoZ+I8Omnn6Jz584wNDREUFAQ8vPzOzDj55dKpUJ4eDhcXV1haGiIV155Bf/85z9F13ngerbul19+wVtvvQUHBwdIJBLExcWJ+jWpX2lpKUJCQmBmZgYLCwtMmzYN9+/f126ixNpk3759pKenR9988w3l5OTQjBkzyMLCgm7fvt3RqT33hg0bRlFRUZSdnU1ZWVk0cuRIcnJyovv37wsxM2fOJEdHRzp+/DidPXuWBg0aRIMHD+7ArF8M6enp5OLiQv369aP58+cL7VzPtiktLSVnZ2d6//33KS0tjQoKCujYsWN0+fJlIWbNmjVkbm5OcXFxdO7cORo9ejS5urrSgwcPOjDz51NERARZWVlRfHw8FRYW0oEDB8jExIQ2b94sxHA9W3fkyBFavnw5HTx4kABQbGysqF+T+g0fPpz69+9Pp0+fpuTkZOrWrRtNnDhRq3nyZKKNvL29ac6cOcJ9lUpFDg4OtHr16g7M6sV0584dAkBJSUlERFRWVkYymYwOHDggxFy4cIEA0KlTpzoqzedeZWUlubu7U0JCAgUEBAiTCa5n233yySf0l7/8pcV+tVpN9vb2tG7dOqGtrKyM9PX16fvvv38WKb5QRo0aRVOnThW1jR07lkJCQoiI69lWj04mNKlfbm4uAaAzZ84IMUePHiWJREJFRUVay413c7SBUqlERkYGgoKChDYdHR0EBQXh1KlTHZjZi6m8vBwA0KlTJwBARkYG6urqRPXt2bMnnJycuL6tmDNnDkaNGiWqG8D1fBKHDx/GwIEDMX78eNja2sLT0xM7d+4U+gsLC6FQKEQ1NTc3h4+PD9e0GYMHD8bx48eRl5cHADh37hxSUlIwYsQIAFzP9tKkfqdOnYKFhQUGDhwoxAQFBUFHRwdpaWlay+Wlv9CXNhUXF0OlUsHOzk7Ubmdnh4sXL3ZQVi8mtVqNsLAw+Pn5oU+fPgAAhUIBPT09WFhYiGLt7OygUCg6IMvn3759+5CZmYkzZ8406eN6tl1BQQG2bt2KhQsXYtmyZThz5gzmzZsHPT09hIaGCnVr7jOAa9rUkiVLUFFRgZ49e0JXVxcqlQoREREICQkBAK5nO2lSP4VCAVtbW1G/VCpFp06dtFpjnkywDjFnzhxkZ2cjJSWlo1N5Yd24cQPz589HQkICDAwMOjqdl4JarcbAgQOxatUqAICnpyeys7Oxbds2hIaGdnB2L579+/cjJiYGe/fuRe/evZGVlYWwsDA4ODhwPV8yvJujDaytraGrq9tkNfzt27dhb2/fQVm9eObOnYv4+HicPHlSdHl4e3t7KJVKlJWVieK5vs3LyMjAnTt3MGDAAEilUkilUiQlJWHLli2QSqWws7PjerZR586d0atXL1Gbh4cHrl+/DgBC3fgzQDOLFy/GkiVL8M4776Bv376YPHkyFixYgNWrVwPgeraXJvWzt7fHnTt3RP319fUoLS3Vao15MtEGenp68PLywvHjx4U2tVqN48ePw9fXtwMzezEQEebOnYvY2FicOHECrq6uon4vLy/IZDJRfS9duoTr169zfZsxZMgQ/Pbbb8jKyhJuAwcOREhIiPBvrmfb+Pn5NTlcOS8vD87OzgAAV1dX2Nvbi2paUVGBtLQ0rmkzqquroaMj/prR1dWFWq0GwPVsL03q5+vri7KyMmRkZAgxJ06cgFqtho+Pj/aS0dpSzj+Jffv2kb6+PkVHR1Nubi598MEHZGFhQQqFoqNTe+7NmjWLzM3NKTExkW7duiXcqqurhZiZM2eSk5MTnThxgs6ePUu+vr7k6+vbgVm/WB4+moOI69lW6enpJJVKKSIigvLz8ykmJoaMjIxoz549QsyaNWvIwsKCDh06ROfPn6fg4GA+lLEFoaGh1KVLF+HQ0IMHD5K1tTV9/PHHQgzXs3WVlZUkl8tJLpcTAIqMjCS5XE7Xrl0jIs3qN3z4cPL09KS0tDRKSUkhd3d3PjT0efDFF1+Qk5MT6enpkbe3N50+fbqjU3ohAGj2FhUVJcQ8ePCAZs+eTZaWlmRkZER/+9vf6NatWx2X9Avm0ckE17PtfvzxR+rTpw/p6+tTz549aceOHaJ+tVpN4eHhZGdnR/r6+jRkyBC6dOlSB2X7fKuoqKD58+eTk5MTGRgYkJubGy1fvpxqa2uFGK5n606ePNns52ZoaCgRaVa/kpISmjhxIpmYmJCZmRlNmTKFKisrtZonX4KcMcYYY+3CayYYY4wx1i48mWCMMcZYu/BkgjHGGGPtwpMJxhhjjLULTyYYY4wx1i48mWCMMcZYu/BkgjHGGGPtwpMJxhhjjLULTyYYYx1CIpEgLi5O4/j3338fY8aMeWr5POzh3K5evQqJRIKsrKwnHq+kpAS2tra4evXqE49RXFwMW1tb/P777088BmNPC08mGNOSu3fvYtasWXBycoK+vj7s7e0xbNgwpKamCjFt/QLtCNHR0ZBIJPDw8GjSd+DAAUgkEri4uDz7xDRARNixYwd8fHxgYmICCwsLDBw4EJs2bUJ1dXWH5RUREYHg4GChbqWlpXjrrbdgYmICT09PyOVyUfycOXOwYcMGUZu1tTXee+89rFix4lmlzZjGeDLBmJaMGzcOcrkcu3fvRl5eHg4fPozAwECUlJS0aRylUvmUMtScsbEx7ty5g1OnTonad+3aBScnpw7K6vEmT56MsLAwBAcH4+TJk8jKykJ4eDgOHTqEn376qUNyqq6uxq5duzBt2jShLSIiApWVlcjMzERgYCBmzJgh9J0+fRppaWkICwtrMtaUKVMQExOD0tLSZ5E6Y5rT6pU+GPuTunfvHgGgxMTEFmOcnZ1FF+pxdnYmIqIVK1ZQ//79aefOneTi4kISiYSIiK5du0ajR48mY2NjMjU1pfHjx4uuTtv4uG+//ZacnZ3JzMyMJkyYQBUVFUJMRUUFTZo0iYyMjMje3p4iIyObXAzsUVFRUWRubk5z586l6dOnC+03btwgfX19WrJkiZB7o3//+9/k5uZGMpmMunfvTt9++62oPy8vj15//XXS19cnDw8P+umnnwgAxcbGCjHXr1+n8ePHk7m5OVlaWtLo0aOpsLBQ6A8NDaXg4OAW8/7hhx8IAMXFxTXpU6vVVFZWRkQNVwYNCgoiKysrMjMzI39/f8rIyBDFP5xbYWEhASC5XE5ERKWlpTRp0iSytrYmAwMD6tatG33zzTct5nXgwAGysbERtY0YMYK2bt1KRES5ublkZGRERERKpZL69+9PZ86caXE8V1dX+vrrr1vsZ6wj8JYJxrTAxMQEJiYmiIuLQ21tbbMxZ86cAQBERUXh1q1bwn0AuHz5Mv773//i4MGDyMrKglqtRnBwMEpLS5GUlISEhAQUFBRgwoQJojGvXLmCuLg4xMfHIz4+HklJSVizZo3Qv3DhQqSmpuLw4cNISEhAcnIyMjMzNXpNU6dOxf79+4XdA9HR0Rg+fDjs7OxEcbGxsZg/fz4WLVqE7OxsfPjhh5gyZQpOnjwJAFCr1Rg7diz09PSQlpaGbdu24ZNPPhGNUVdXh2HDhsHU1BTJyclITU2FiYkJhg8frvGWmpiYGPTo0QPBwcFN+iQSCczNzQEAlZWVCA0NRUpKCk6fPg13d3eMHDkSlZWVGj1PeHg4cnNzcfToUVy4cAFbt26FtbV1i/HJycnw8vIStfXv3x8nTpxAfX09jh07hn79+gEA1q5di8DAQAwcOLDF8by9vZGcnKxRrow9Mx09m2HsZfGf//yHLC0tycDAgAYPHkxLly6lc+fOiWLwyK9xooYtDDKZjO7cuSO0/fTTT6Srq0vXr18X2nJycggApaenC48zMjISbYlYvHgx+fj4EFHDVgmZTEYHDhwQ+svKysjIyEijLRNERK+++irt3r2b1Go1vfLKK3To0CHauHGjaMvE4MGDacaMGaIxxo8fTyNHjiQiomPHjpFUKqWioiKh/+jRo6JafPfdd9SjRw9Sq9VCTG1tLRkaGtKxY8eI6PFbJjw8PGj06NEt9rdEpVKRqakp/fjjj0IbWtky8dZbb9GUKVM0Hj84OJimTp0qaisrK6OJEyeSk5MT+fv7U05ODuXl5ZG7uzsVFxfThx9+SK6urjR+/Hhhi0qjBQsWUGBgYJtfJ2NPE2+ZYExLxo0bh5s3b+Lw4cMYPnw4EhMTMWDAAERHRz/2sc7OzrCxsRHuX7hwAY6OjnB0dBTaevXqBQsLC1y4cEFoc3FxgampqXC/c+fOuHPnDgCgoKAAdXV18Pb2FvrNzc3Ro0cPjV/T1KlTERUVhaSkJFRVVWHkyJFNYi5cuAA/Pz9Rm5+fn5Bn42txcHAQ+n19fUXx586dw+XLl2Fqaips5enUqRNqampw5coVjXIlIo3ibt++jRkzZsDd3R3m5uYwMzPD/fv3cf36dY0eP2vWLOzbtw+vvvoqPv74Y/z666+txj948AAGBgaiNnNzc+zduxfXrl1DUlISevXqhQ8//BDr1q1DTEwMCgoKcOnSJRgZGWHlypWixxoaGnboYlLGmsOTCca0yMDAAEOHDkV4eDh+/fVXvP/++xqtvjc2Nn6i55PJZKL7EokEarX6icZqTkhICE6fPo3PPvsMkydPhlQq1drYD7t//z68vLyQlZUluuXl5WHSpEkajdG9e3dcvHjxsXGhoaHIysrC5s2b8euvvyIrKwtWVlYa704ZMWIErl27hgULFuDmzZsYMmQIPvrooxbjra2tce/evVbHjIqKgoWFBYKDg5GYmIgxY8ZAJpNh/PjxSExMFMWWlpaKJp6MPQ94MsHYU9SrVy9UVVUJ92UyGVQq1WMf5+HhgRs3buDGjRtCW25uLsrKytCrVy+NntvNzQ0ymUy0NqO8vBx5eXka59+pUyeMHj0aSUlJmDp1aou5Pnz4KwCkpqYKeTa+llu3bgn9p0+fFsUPGDAA+fn5sLW1Rbdu3US3xrUOjzNp0iTk5eXh0KFDTfqICOXl5UJu8+bNw8iRI9G7d2/o6+ujuLhYo+doZGNjg9DQUOzZswebNm3Cjh07Woz19PREbm5ui/13797FypUr8cUXXwAAVCoV6urqADSsJXn0/ZKdnQ1PT8825cvY08aTCca0oKSkBG+88Qb27NmD8+fPo7CwEAcOHMDatWtFCwJdXFxw/PhxKBSKVn+tBgUFoW/fvggJCUFmZibS09Px3nvvISAgoNXFeQ8zNTVFaGgoFi9ejJMnTyInJwfTpk2Djo4OJBKJxq8tOjoaxcXF6NmzZ7P9ixcvRnR0NLZu3Yr8/HxERkbi4MGDwq/1oKAgdO/eHaGhoTh37hySk5OxfPly0RghISGwtrZGcHAwkpOTUVhYiMTERMybN0/jkzS9/fbbmDBhAiZOnIhVq1bh7NmzuHbtGuLj4xEUFCQsCHV3d8d3332HCxcuIC0tDSEhITA0NNS4Hp9++ikOHTqEy5cvIycnB/Hx8c2ek6PRsGHDkJOT0+LfOywsDIsWLUKXLl0ANOwiasxvx44dol1I1dXVyMjIwJtvvqlxvow9CzyZYEwLTExM4OPjg40bN8Lf3x99+vRBeHg4ZsyYgS+//FKI27BhAxISEuDo6Njqr0uJRIJDhw7B0tIS/v7+CAoKgpubG3744Yc25RUZGQlfX1/89a9/RVBQEPz8/ODh4dFkH35rDA0NYWVl1WL/mDFjsHnzZqxfvx69e/fG9u3bERUVhcDAQACAjo4OYmNj8eDBA3h7e2P69OmIiIgQjWFkZIRffvkFTk5OGDt2LDw8PDBt2jTU1NTAzMxMozwlEgn27t2LyMhIxMXFISAgAP369cNnn32G4OBgDBs2DEDDuTLu3buHAQMGYPLkyZg3bx5sbW01roeenh6WLl2Kfv36wd/fH7q6uti3b1+L8X379sWAAQOwf//+Jn3Hjh3D5cuXMXv2bKFt7ty5cHNzg4+PD5RKpWg32aFDh+Dk5ITXX39d43wZexYkpOmqJcbYC6+qqgpdunTBhg0bRCdRYk/X//73PyxevBjZ2dnQ0Xny33CDBg3CvHnzNF5Hwtiz8nRWUzHGngtyuRwXL16Et7c3ysvLhSMDmjsXA3t6Ro0ahfz8fBQVFYmO0GmL4uJijB07FhMnTtRydoy1H2+ZYOwlJpfLMX36dFy6dAl6enrw8vJCZGQk+vbt29GpMcZeIjyZYIwxxli78AJMxhhjjLULTyYYY4wx1i48mWCMMcZYu/BkgjHGGGPtwpMJxhhjjLULTyYYY4wx1i48mWCMMcZYu/BkgjHGGGPt8n9EZXagoj3zfgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import Image, display\n",
    "\n",
    "# Display full plot saved in the following path\n",
    "image_path = \"/home/ray/default/RouteLLM/gsm8k.png\"\n",
    "display(Image(filename=image_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This plot illustrates that as we relax the cost constraints (i.e., increase the percentage of GPT-4 calls), the performance improves. While the performance of a random router improves linearly with cost, our router achieves significantly better results at each cost level."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "In this tutorial, we have successfully built and evaluated a finetuned-LLM router. We generated synthetic labeled data using the LLM-as-a-judge method to train the model, finetuned an LLM classifier using Anyscale's API, and conducted offline evaluation on a standard benchmark-- demonstrating that our model is effective in out-of-domain generalization."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}