JobDomainPrediction_NLP.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "JobDomainPrediction_NLP.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "72b3b3a25ab4452ebee962bf0f627b0a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_f41cc55d32974a788b6aabb78f9f8eac",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_80d6c9bafe3b42bc94b9a29715e1e3c0",
              "IPY_MODEL_a59ce2ff4f284bc09602866cf243c4bf"
            ]
          }
        },
        "5ba89c16444943d9b0f16353d73cf95c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_750522277df847ce97c823761f04f474",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_cd749f2ec41c4358a1b6ab259deb9895",
              "IPY_MODEL_8c29a3e79be140deb1a0ffd98bc0869e"
            ]
          }
        },
        "633004165163472e80dd7ea899248a48": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_f2a46fc98ad94ca6895ebb1f82a64567",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_164e25d413804caf97681358bcf76c87",
              "IPY_MODEL_5ab351a64b0042e4b16ff6087a27ea26"
            ]
          }
        }
      }
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/AQLT/NLP/blob/main/JobDomainPrediction_NLP.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "unKdmyC8ldp2"
      },
      "source": [
        "# Guessing the domain of a job from its description"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oA1wLbNX7EDQ"
      },
      "source": [
        "Pôle Emploi is the French governmental agency which registers unemployed people, helps them find jobs and provides them with financial aid. To offer suitable jobs, they use a classification list of jobs called the “ROME” nomenclature. A ROME code corresponds to a job and the 532 jobs of the database are classified in 110 professional domains (“domaine professionnel”) and 14 major domains (“grands domaines”). Each job also contains many denominations (which are many names which refers to the same ROME code).\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "For this project, we tried to answer to the following research question: **Using Natural Language Processing modeling, is it possible to properly guess the domain of a job simply by reading its description?** \n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "To answer this question, we used two datasets provided by **Pôle Emploi** and **web scraped** additional information from **Onisep** (another governmental agency which provides students, parents and teachers information about jobs and training courses). The final dataset is composed of **1 247 descriptions** belonging to **14 different classes** (major domains). Given the limited size of the database used, we only tried here to guess the major domain classification.\n",
        "\n",
        "1.\tThe French official jobs’ taxonomy by Pôle Emploi (https://www.data.gouv.fr/fr/datasets/repertoire-operationnel-des-metiers-et-des-emplois-rome/ “Arborescence principale” file)\n",
        "2.\tThe description of each job\n",
        "\n",
        "  a. provided by Pôle emploi (https://www.pole-emploi.org/opendata/repertoire-operationnel-des-meti.html?type=article > Toutes les données du ROME > CSV >\n",
        "unix_texte_v346_utf8.csv)\n",
        "   + Example of content : https://candidat.pole-emploi.fr/marche-du-travail/fichemetierrome?codeRome=E1301\n",
        "\n",
        "  b.\t provided by Onisep (https://data.laregion.fr/explore/dataset/onisep-metiers-onisep/table/ )\n",
        " + Example of content : https://www.onisep.fr/Ressources/Univers-Metier/Metiers/conducteur-conductrice-de-machine-a-imprimer (frame : \"En quoi consiste ce métier ?\")\n",
        "\n",
        "\n",
        "*Example:*  The “métier” (job) \"Conduite de machines d'impression\" (E1301) belongs to the “domaine professionnel” called “industries graphiques” (E13) and to the “grand domaine” \"Communication, Média et Multimédia\" (E). It has many “appellations” (denominations): Conducteur / Conductrice de machines à imprimer offset (13062), Conducteur / Conductrice de machines d'impression (13088), and many others. \n",
        "The job “Conduite de machines d'impression” has the following descriptions:\n",
        "\n",
        "    “Règle, surveille et régule une machine à imprimer simple ou complexe (offset,\n",
        "     héliogravure, typogravure, ...), sur divers supports (papier, carton, matière plastique\n",
        "     tissu, métal, ...), selon les règles de sécurité et les impératifs de production.\n",
        "     Effectue le contrôle de conformité des produits imprimés et la maintenance de premier\n",
        "     niveau des machines et équipements. Peut coordonner une équipe.”\n",
        "**(Source : Pôle Emploi)**\n",
        "\n",
        "    “Préparer la machine pour l'impression, effectuer les réglages, réaliser des essais,\n",
        "    lancer l'impression de journaux, magazines, étiquettes, affiches... sont autant d'opérations \n",
        "    orchestrées par le conducteur de machines à imprimer. À l'intérieur d'une usine, il est \n",
        "    chargé de piloter une machine de production. Garant de la qualité d'impression, ce \n",
        "    professionnel travaille sur des machines plus ou moins complexes : machines de 1 à 4 couleurs \n",
        "    et plus, à commandes numériques, sérigraphiques, rotatives... ou offset, la \n",
        "    technique d'impression la plus répandue. Il examine les instructions du dossier de \n",
        "    fabrication. Il prépare les encres, le papier, effectue les réglages et procède à des \n",
        "    essais jusqu'à l'obtention du \"bon à rouler\". Ensuite, il lance l'impression et contrôle la \n",
        "    conformité du produit imprimé. En cas de problème, il intervient rapidement, car la \n",
        "    production ne doit surtout pas s'arrêter ! Souvent des équipes se relaient en 3 × 8 heures \n",
        "    ou en 5 × 8 heures. Le conducteur de machines à imprimer obéit à des règles strictes de \n",
        "    sécurité. Le respect des normes européennes en matière d'environnement est également crucial.” \n",
        "**(Source : Onisep)**\n",
        "\n",
        "To predict the different major domains, we split the datasets into 2 sets (training and test sets) and performed **9 models of Sequence Classification** on the training set using `Pytorch`. Once fit, we **evaluated** the model on the test set (unseen jobs and denominations) using F1 scores per class. \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XXQThGHdN2E5"
      },
      "source": [
        "# 0 Downloading the database [optional]\n",
        "\n",
        "All the data are gathered in a gist repository. **You don't need to run the code** of this section where we :\n",
        "\n",
        "1. Download description from the Onisep database with webscrapping (https://data.laregion.fr/explore/dataset/onisep-metiers-onisep/table/)\n",
        "\n",
        "2. Shape Pole Emploi databases (https://www.pole-emploi.org/opendata/repertoire-operationnel-des-meti.html?type=article)\n",
        "\n",
        "Data : https://www.data.gouv.fr/fr/datasets/repertoire-operationnel-des-metiers-et-des-emplois-rome/#_\n",
        "\n",
        "Link onisep : https://data.laregion.fr/explore/dataset/onisep-metiers-onisep/table/\n",
        "\n",
        "Link ROME description : \n",
        "https://candidat.pole-emploi.fr/marche-du-travail/fichemetierrome?codeRome=E1301\n",
        "\n",
        "[NOT USED] Link between ROME and other nomenclatures (PCS for example) : https://www.pole-emploi.org/opendata/repertoire-operationnel-des-meti.html?type=article\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bRG15JGFBGcS"
      },
      "source": [
        "## 0.1 Onisep (webscraping)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7ChdiVrtDBOQ"
      },
      "source": [
        "import pandas as pd\n",
        "import re\n",
        "import requests\n",
        "from bs4 import BeautifulSoup"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RuTli7w1GCHV"
      },
      "source": [
        "First we download a file containing the links to webscrap : 1 link per job or more precisely be \"appellation\"."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mzdsSo0OiCbr",
        "outputId": "426610c1-1c40-4551-e46b-b6d29aadee43"
      },
      "source": [
        "!wget 'https://data.laregion.fr/explore/dataset/onisep-metiers-onisep/download/?format=csv&timezone=Europe/Berlin&lang=fr&use_labels_for_header=true&csv_separator=%3B' -O onisep.csv"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-04-14 15:56:36--  https://data.laregion.fr/explore/dataset/onisep-metiers-onisep/download/?format=csv&timezone=Europe/Berlin&lang=fr&use_labels_for_header=true&csv_separator=%3B\n",
            "Resolving data.laregion.fr (data.laregion.fr)... 171.33.103.241, 109.232.232.161\n",
            "Connecting to data.laregion.fr (data.laregion.fr)|171.33.103.241|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: unspecified [application/csv]\n",
            "Saving to: ‘onisep.csv’\n",
            "\n",
            "onisep.csv              [     <=>            ] 460.52K   449KB/s    in 1.0s    \n",
            "\n",
            "2021-04-14 15:56:40 (449 KB/s) - ‘onisep.csv’ saved [471572]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c-N4kpLkGnmT"
      },
      "source": [
        "We shape this database."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 864
        },
        "id": "ovo42qLIhsn3",
        "outputId": "55d0baa1-0de5-459f-f2ed-f4943e5d225a"
      },
      "source": [
        "onisep_table = pd.read_csv(\"onisep.csv\",sep=\";\")\n",
        "onisep_table = onisep_table.dropna(subset=['code ROME', 'lien site onisep.fr'])\n",
        "onisep_table = onisep_table[[\"libellé métier\", \"lien site onisep.fr\", \"GFE\", \"domaine/sous-domaine ONISEP\",\"code ROME\"]]\n",
        "onisep_table = onisep_table.rename(columns={\"libellé métier\": \"metier\",\n",
        "                                            \"lien site onisep.fr\": \"url\",\n",
        "                                            \"domaine/sous-domaine ONISEP\": \"domaine_ONISEP\",\n",
        "                                            \"code ROME\": \"rome_code\"})\n",
        "onisep_table[\"sous_domaine_ONISEP\"] = onisep_table[\"domaine_ONISEP\"].apply(lambda x: re.sub(\".*/\",\"\",x))\n",
        "onisep_table[\"domaine_ONISEP\"] = onisep_table[\"domaine_ONISEP\"].apply(lambda x: re.sub(\"/.*\",\"\",x))\n",
        "onisep_table = onisep_table.drop_duplicates(subset=['url']) # some url are duplicated\n",
        "onisep_table"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>metier</th>\n",
              "      <th>url</th>\n",
              "      <th>GFE</th>\n",
              "      <th>domaine_ONISEP</th>\n",
              "      <th>rome_code</th>\n",
              "      <th>sous_domaine_ONISEP</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>architecte</td>\n",
              "      <td>http://www.onisep.fr/http/redirection/metier/i...</td>\n",
              "      <td>C : Bâtiment</td>\n",
              "      <td>construction - architecture - travaux publics</td>\n",
              "      <td>F1101</td>\n",
              "      <td>architecture</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>géomètre-topographe</td>\n",
              "      <td>http://www.onisep.fr/http/redirection/metier/i...</td>\n",
              "      <td>C : Bâtiment</td>\n",
              "      <td>construction - architecture - travaux publics</td>\n",
              "      <td>F1108</td>\n",
              "      <td>sciences de la Terre</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>technicien / technicienne de maintenance en gé...</td>\n",
              "      <td>http://www.onisep.fr/http/redirection/metier/i...</td>\n",
              "      <td>H : Electricité, énergie, électronique</td>\n",
              "      <td>construction - architecture - travaux publics</td>\n",
              "      <td>I1308</td>\n",
              "      <td>énergies</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>menuisier / menuisière</td>\n",
              "      <td>http://www.onisep.fr/http/redirection/metier/i...</td>\n",
              "      <td>M : Travail du bois</td>\n",
              "      <td>construction - architecture - travaux publics</td>\n",
              "      <td>H2202</td>\n",
              "      <td>bois</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>acheteur / acheteuse</td>\n",
              "      <td>http://www.onisep.fr/http/redirection/metier/i...</td>\n",
              "      <td>Q : Commerce, gestion, finances</td>\n",
              "      <td>commerce, marketing, vente</td>\n",
              "      <td>M1101</td>\n",
              "      <td>grande distribution et petits commerces</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1131</th>\n",
              "      <td>pédologue</td>\n",
              "      <td>http://www.onisep.fr/http/redirection/metier/i...</td>\n",
              "      <td>T : Environnement, nettoyage, sécurité</td>\n",
              "      <td>sciences</td>\n",
              "      <td>F1105</td>\n",
              "      <td>sciences de la Terre</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1136</th>\n",
              "      <td>statisticien industriel / statisticienne indus...</td>\n",
              "      <td>http://www.onisep.fr/http/redirection/metier/i...</td>\n",
              "      <td>P : Gestion et traitement de l'information (in...</td>\n",
              "      <td>sciences</td>\n",
              "      <td>M1403</td>\n",
              "      <td>mathématiques</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1138</th>\n",
              "      <td>ingénieur / ingénieure en caractérisation des ...</td>\n",
              "      <td>http://www.onisep.fr/http/redirection/metier/i...</td>\n",
              "      <td>J : Transformation des matériaux, procédés (bi...</td>\n",
              "      <td>matières premières, fabrication, industries</td>\n",
              "      <td>H1206</td>\n",
              "      <td>matériaux</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1140</th>\n",
              "      <td>metteur / metteuse en scène</td>\n",
              "      <td>http://www.onisep.fr/http/redirection/metier/i...</td>\n",
              "      <td>V : Arts, spectacle, audiovisuel</td>\n",
              "      <td>arts, culture, artisanat</td>\n",
              "      <td>L1301</td>\n",
              "      <td>arts du spectacle</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1141</th>\n",
              "      <td>expert / experte en assurances</td>\n",
              "      <td>http://www.onisep.fr/http/redirection/metier/i...</td>\n",
              "      <td>Q : Commerce, gestion, finances</td>\n",
              "      <td>banque - assurances - immobilier</td>\n",
              "      <td>C1106</td>\n",
              "      <td>assurances</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>715 rows × 6 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                 metier  ...                      sous_domaine_ONISEP\n",
              "0                                            architecte  ...                             architecture\n",
              "1                                   géomètre-topographe  ...                     sciences de la Terre\n",
              "2     technicien / technicienne de maintenance en gé...  ...                                 énergies\n",
              "3                                menuisier / menuisière  ...                                     bois\n",
              "4                                  acheteur / acheteuse  ...  grande distribution et petits commerces\n",
              "...                                                 ...  ...                                      ...\n",
              "1131                                          pédologue  ...                     sciences de la Terre\n",
              "1136  statisticien industriel / statisticienne indus...  ...                            mathématiques\n",
              "1138  ingénieur / ingénieure en caractérisation des ...  ...                                matériaux\n",
              "1140                        metteur / metteuse en scène  ...                        arts du spectacle\n",
              "1141                     expert / experte en assurances  ...                               assurances\n",
              "\n",
              "[715 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GkNElpWlGugC"
      },
      "source": [
        "And we perform a webscraping to get the description of each job."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z1ICbW1ykthd"
      },
      "source": [
        "#20 minutes to run\n",
        "def dl_description(url):\n",
        "  page = requests.get(url)\n",
        "  soup = BeautifulSoup(page.content, 'html.parser')\n",
        "  results = soup.find(id=\"en-quoi-consiste-ce-metier\")\n",
        "  paragraph = results.find_all('p')\n",
        "  return (\"\\n\".join([text.get_text() for text in paragraph]))\n",
        "onisep_table[\"description\"] = onisep_table.url.apply(dl_description)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wo16lMAkG57Q"
      },
      "source": [
        "We tidy the database."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 912
        },
        "id": "bs9Fws3Ntb54",
        "outputId": "030ae352-35bd-4327-ac82-a7413f5d2960"
      },
      "source": [
        "onisep_table[\"description\"] = onisep_table[\"description\"].apply(lambda x: re.sub(\"\\xa0\", \" \", x))\n",
        "onisep_table = onisep_table.rename(columns={\"onisep_description\": \"description\"}).drop(columns=[\"url\"])\n",
        "onisep_table.to_csv(\"onisep_description.csv\", index=False)\n",
        "#we have the database in csv and we put it on gist\n",
        "onisep_table"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>metier</th>\n",
              "      <th>GFE</th>\n",
              "      <th>domaine_ONISEP</th>\n",
              "      <th>rome_code</th>\n",
              "      <th>sous_domaine_ONISEP</th>\n",
              "      <th>description</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>architecte</td>\n",
              "      <td>C : Bâtiment</td>\n",
              "      <td>construction - architecture - travaux publics</td>\n",
              "      <td>F1101</td>\n",
              "      <td>architecture</td>\n",
              "      <td>Construction d'un immeuble, réaménagement d'un...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>géomètre-topographe</td>\n",
              "      <td>C : Bâtiment</td>\n",
              "      <td>construction - architecture - travaux publics</td>\n",
              "      <td>F1108</td>\n",
              "      <td>sciences de la Terre</td>\n",
              "      <td>L'oeil rivé dans le viseur de son tachéomètre,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>technicien / technicienne de maintenance en gé...</td>\n",
              "      <td>H : Electricité, énergie, électronique</td>\n",
              "      <td>construction - architecture - travaux publics</td>\n",
              "      <td>I1308</td>\n",
              "      <td>énergies</td>\n",
              "      <td>Professionnel de l'ambiance et du confort, le ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>menuisier / menuisière</td>\n",
              "      <td>M : Travail du bois</td>\n",
              "      <td>construction - architecture - travaux publics</td>\n",
              "      <td>H2202</td>\n",
              "      <td>bois</td>\n",
              "      <td>Travailler le bois, lui donner une seconde vie...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>acheteur / acheteuse</td>\n",
              "      <td>Q : Commerce, gestion, finances</td>\n",
              "      <td>commerce, marketing, vente</td>\n",
              "      <td>M1101</td>\n",
              "      <td>grande distribution et petits commerces</td>\n",
              "      <td>Le rôle d'un acheteur est d'acheter des produi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1131</th>\n",
              "      <td>pédologue</td>\n",
              "      <td>T : Environnement, nettoyage, sécurité</td>\n",
              "      <td>sciences</td>\n",
              "      <td>F1105</td>\n",
              "      <td>sciences de la Terre</td>\n",
              "      <td>Le pédologue est un scientifique spécialiste d...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1136</th>\n",
              "      <td>statisticien industriel / statisticienne indus...</td>\n",
              "      <td>P : Gestion et traitement de l'information (in...</td>\n",
              "      <td>sciences</td>\n",
              "      <td>M1403</td>\n",
              "      <td>mathématiques</td>\n",
              "      <td>Prévoir la durée de vie d'un produit industrie...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1138</th>\n",
              "      <td>ingénieur / ingénieure en caractérisation des ...</td>\n",
              "      <td>J : Transformation des matériaux, procédés (bi...</td>\n",
              "      <td>matières premières, fabrication, industries</td>\n",
              "      <td>H1206</td>\n",
              "      <td>matériaux</td>\n",
              "      <td>L'ingénieur en caractérisation des matériaux c...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1140</th>\n",
              "      <td>metteur / metteuse en scène</td>\n",
              "      <td>V : Arts, spectacle, audiovisuel</td>\n",
              "      <td>arts, culture, artisanat</td>\n",
              "      <td>L1301</td>\n",
              "      <td>arts du spectacle</td>\n",
              "      <td>Quand les 3 coups résonnent dans la salle et q...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1141</th>\n",
              "      <td>expert / experte en assurances</td>\n",
              "      <td>Q : Commerce, gestion, finances</td>\n",
              "      <td>banque - assurances - immobilier</td>\n",
              "      <td>C1106</td>\n",
              "      <td>assurances</td>\n",
              "      <td>Conseiller technique et souvent arbitre, l'exp...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>715 rows × 6 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                 metier  ...                                        description\n",
              "0                                            architecte  ...  Construction d'un immeuble, réaménagement d'un...\n",
              "1                                   géomètre-topographe  ...  L'oeil rivé dans le viseur de son tachéomètre,...\n",
              "2     technicien / technicienne de maintenance en gé...  ...  Professionnel de l'ambiance et du confort, le ...\n",
              "3                                menuisier / menuisière  ...  Travailler le bois, lui donner une seconde vie...\n",
              "4                                  acheteur / acheteuse  ...  Le rôle d'un acheteur est d'acheter des produi...\n",
              "...                                                 ...  ...                                                ...\n",
              "1131                                          pédologue  ...  Le pédologue est un scientifique spécialiste d...\n",
              "1136  statisticien industriel / statisticienne indus...  ...  Prévoir la durée de vie d'un produit industrie...\n",
              "1138  ingénieur / ingénieure en caractérisation des ...  ...  L'ingénieur en caractérisation des matériaux c...\n",
              "1140                        metteur / metteuse en scène  ...  Quand les 3 coups résonnent dans la salle et q...\n",
              "1141                     expert / experte en assurances  ...  Conseiller technique et souvent arbitre, l'exp...\n",
              "\n",
              "[715 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ANuV5ccwBMqi"
      },
      "source": [
        "## 0.2 Pole Emploi"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j0Zj6izxPRsp"
      },
      "source": [
        "import pandas as pd\n",
        "import re"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zyTmTYAUHWQF"
      },
      "source": [
        "First we download the files from the Pole Emploi API."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6z09KLJgfZ6f"
      },
      "source": [
        "!wget https://api.emploi-store.fr/api/docs/romeopen/REF_ROME_CSV/1/RefRomeCsv.zip -O metadata.zip\n",
        "!unzip metadata.zip\n",
        "!rm metadata.zip"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xLL3VA6ECwh2"
      },
      "source": [
        "In the ROME database, there are two tree views:\n",
        "\n",
        "- The principal one (`code_type_referentiel = 6`) organised in 4 levels:\n",
        "  * 14 major fields (*grands domaines*) => **Our future classes**\n",
        "  * 110 professional fields (*domaines professionnels*)\n",
        "  * 532 ROME forms (*fiches ROME*)\n",
        "  * 11 112 job names (*appellations métier*)\n",
        "- The thematic one (`code_type_referentiel = 7`) with 17 main fields associated to 509 ROME forms: not all ROME forms are included in this tree and some of them are in several main fields at the same time (in total, 295 different ROME forms are used). That's why we will mainly focus on major the principal tree.\n",
        "\n",
        "In the Pole Emploi database we have exhaustive description of the ROME jobs (532 observations), whereas in the Onisep database we have non exhaustive description of 715 job appellations (with the associated ROME forms, 354 ROME forms are observed)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dzJMy_sOIA7d"
      },
      "source": [
        "We now load and tidy the different databases."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PMKsWu7CHbBl"
      },
      "source": [
        "arbo = pd.read_csv(\"unix_item_arborescence_v346_utf8.csv\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "UTB58cvZFLVV",
        "outputId": "64b50539-bfd8-408f-9933-68ae39c045ea"
      },
      "source": [
        "thematic_tree = arbo[(arbo[\"code_type_noeud\"]==3)&(arbo[\"code_type_referentiel\"]==7)]\n",
        "thematic_tree = thematic_tree.rename(columns={\"code_noeud\": \"rome_code\",\n",
        "                                              \"code_pere\": \"thematic_code\",\n",
        "                                              \"libelle_item_arbo\": \"rome_title\"})\n",
        "thematic_tree = thematic_tree.drop(columns = ['code_ogr', 'code_item_arbo_associe', 'code_type_noeud',\n",
        "       'libelle_type_noeud', 'statut',\"code_type_referentiel\"])\n",
        "print(thematic_tree)\n",
        "th_tree_desc = arbo[(arbo[\"code_type_noeud\"]==2)&(arbo[\"code_type_referentiel\"]==7)]\n",
        "th_tree_desc = th_tree_desc.drop(columns = ['code_ogr', 'code_item_arbo_associe', 'code_type_noeud',\n",
        "       'libelle_type_noeud', 'code_pere',\"statut\",\"code_type_referentiel\"])\n",
        "th_tree_desc = th_tree_desc.rename(columns={\"code_noeud\": \"thematic_code\",\n",
        "                                            \"libelle_item_arbo\": \"thematic_title\"})\n",
        "print(th_tree_desc)\n",
        "thematic_tree = pd.merge(th_tree_desc,thematic_tree)\n",
        "thematic_tree"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "     thematic_code rome_code                                         rome_title\n",
            "675            001     A1201                            Bûcheronnage et élagage\n",
            "676            001     A1202                     Entretien des espaces naturels\n",
            "677            001     A1203         Aménagement et entretien des espaces verts\n",
            "678            014     A1203         Aménagement et entretien des espaces verts\n",
            "679            003     A1204                   Protection du patrimoine naturel\n",
            "...            ...       ...                                                ...\n",
            "1179           017     N4104                      Courses et livraisons express\n",
            "1180           013     N4104                      Courses et livraisons express\n",
            "1181           014     N4105  Conduite et livraison par tournées sur courte ...\n",
            "1182           017     N4105  Conduite et livraison par tournées sur courte ...\n",
            "1183           013     N4402  Exploitation et manoeuvre des remontées mécani...\n",
            "\n",
            "[509 rows x 3 columns]\n",
            "    thematic_code                                     thematic_title\n",
            "658           001  Métiers de l'environnement et du développement...\n",
            "659           002  Métiers de la défense et de la sécurité publiq...\n",
            "660           003  Métiers du patrimoine et de la restauration d'...\n",
            "661           004               Métiers de l'intelligence économique\n",
            "662           005                            Métiers de la recherche\n",
            "663           006  Métiers de la mer, du nautisme et de la constr...\n",
            "664           007                          Métiers de l'aéronautique\n",
            "665           008          Métiers de la sécurité publique et privée\n",
            "666           009                              Métiers du multimédia\n",
            "667           010  Métiers de l'humanitaire (hors santé et tertia...\n",
            "668           011                               Métiers du nucléaire\n",
            "669           012                         Métiers auprès des enfants\n",
            "670           013      Métiers saisonniers, de vacances / Jobs d'été\n",
            "671           014      Métiers des services à la personne à domicile\n",
            "672           015                Métiers du sport et autour du sport\n",
            "673           016                            Métiers de l'ingénierie\n",
            "674           017  Métiers accessibles sans diplôme et sans expér...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>thematic_code</th>\n",
              "      <th>thematic_title</th>\n",
              "      <th>rome_code</th>\n",
              "      <th>rome_title</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>001</td>\n",
              "      <td>Métiers de l'environnement et du développement...</td>\n",
              "      <td>A1201</td>\n",
              "      <td>Bûcheronnage et élagage</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>001</td>\n",
              "      <td>Métiers de l'environnement et du développement...</td>\n",
              "      <td>A1202</td>\n",
              "      <td>Entretien des espaces naturels</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>001</td>\n",
              "      <td>Métiers de l'environnement et du développement...</td>\n",
              "      <td>A1203</td>\n",
              "      <td>Aménagement et entretien des espaces verts</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>001</td>\n",
              "      <td>Métiers de l'environnement et du développement...</td>\n",
              "      <td>A1204</td>\n",
              "      <td>Protection du patrimoine naturel</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>001</td>\n",
              "      <td>Métiers de l'environnement et du développement...</td>\n",
              "      <td>A1205</td>\n",
              "      <td>Sylviculture</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>504</th>\n",
              "      <td>017</td>\n",
              "      <td>Métiers accessibles sans diplôme et sans expér...</td>\n",
              "      <td>N4101</td>\n",
              "      <td>Conduite de transport de marchandises sur long...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>505</th>\n",
              "      <td>017</td>\n",
              "      <td>Métiers accessibles sans diplôme et sans expér...</td>\n",
              "      <td>N4102</td>\n",
              "      <td>Conduite de transport de particuliers</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>506</th>\n",
              "      <td>017</td>\n",
              "      <td>Métiers accessibles sans diplôme et sans expér...</td>\n",
              "      <td>N4103</td>\n",
              "      <td>Conduite de transport en commun sur route</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>507</th>\n",
              "      <td>017</td>\n",
              "      <td>Métiers accessibles sans diplôme et sans expér...</td>\n",
              "      <td>N4104</td>\n",
              "      <td>Courses et livraisons express</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>508</th>\n",
              "      <td>017</td>\n",
              "      <td>Métiers accessibles sans diplôme et sans expér...</td>\n",
              "      <td>N4105</td>\n",
              "      <td>Conduite et livraison par tournées sur courte ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>509 rows × 4 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "    thematic_code  ...                                         rome_title\n",
              "0             001  ...                            Bûcheronnage et élagage\n",
              "1             001  ...                     Entretien des espaces naturels\n",
              "2             001  ...         Aménagement et entretien des espaces verts\n",
              "3             001  ...                   Protection du patrimoine naturel\n",
              "4             001  ...                                       Sylviculture\n",
              "..            ...  ...                                                ...\n",
              "504           017  ...  Conduite de transport de marchandises sur long...\n",
              "505           017  ...              Conduite de transport de particuliers\n",
              "506           017  ...          Conduite de transport en commun sur route\n",
              "507           017  ...                      Courses et livraisons express\n",
              "508           017  ...  Conduite et livraison par tournées sur courte ...\n",
              "\n",
              "[509 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "20h9GKBJFSF7",
        "outputId": "3150e7cc-70db-4159-d497-a204a011df24"
      },
      "source": [
        "principal_tree = arbo[(arbo[\"code_type_noeud\"]==3)&(arbo[\"code_type_referentiel\"]==6)]\n",
        "principal_tree = principal_tree.rename(columns={\"code_noeud\": \"rome_code\",\n",
        "                                              \"code_pere\": \"professional_code\",\n",
        "                                              \"libelle_item_arbo\": \"rome_title\"})\n",
        "principal_tree = principal_tree.drop(columns = ['code_ogr', 'code_item_arbo_associe', 'code_type_noeud',\n",
        "       'libelle_type_noeud', 'statut',\"code_type_referentiel\"])\n",
        "print(principal_tree)\n",
        "pr_tree_desc = arbo[(arbo[\"code_type_noeud\"]==2)&(arbo[\"code_type_referentiel\"]==6)&(arbo[\"code_pere\"]!=\"Racine AP\")]\n",
        "pr_tree_desc = pr_tree_desc.drop(columns = ['code_ogr', 'code_item_arbo_associe', 'code_type_noeud',\n",
        "       'libelle_type_noeud',\"statut\",\"code_type_referentiel\"])\n",
        "pr_tree_desc = pr_tree_desc.rename(columns={\"code_noeud\": \"professional_code\",\n",
        "                                            \"code_pere\": \"major_code\",\n",
        "                                            \"libelle_item_arbo\": \"professional_title\",})\n",
        "print(pr_tree_desc)\n",
        "maj_tree_desc = arbo[(arbo[\"code_type_noeud\"]==2)&(arbo[\"code_type_referentiel\"]==6)&(arbo[\"code_pere\"]==\"Racine AP\")]\n",
        "maj_tree_desc = maj_tree_desc.drop(columns = ['code_ogr', 'code_item_arbo_associe', 'code_type_noeud',\n",
        "       'libelle_type_noeud',\"statut\",\"code_type_referentiel\", \"code_pere\"])\n",
        "maj_tree_desc = maj_tree_desc.rename(columns={\"code_noeud\": \"major_code\",\n",
        "                                            \"libelle_item_arbo\": \"major_title\"})\n",
        "print(principal_tree)\n",
        "principal_tree = pd.merge(pd.merge(maj_tree_desc,pr_tree_desc),\n",
        "                          principal_tree)\n",
        "principal_tree"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "    professional_code  ...                                         rome_title\n",
            "125               A11  ...          Conduite d'engins agricoles et forestiers\n",
            "126               A12  ...                            Bûcheronnage et élagage\n",
            "127               A12  ...                     Entretien des espaces naturels\n",
            "128               A12  ...         Aménagement et entretien des espaces verts\n",
            "129               A12  ...                   Protection du patrimoine naturel\n",
            "..                ...  ...                                                ...\n",
            "652               N43  ...                                 Conduite sur rails\n",
            "653               N43  ...                  Contrôle des transports en commun\n",
            "654               N44  ...                        Circulation du réseau ferré\n",
            "655               N44  ...  Exploitation et manoeuvre des remontées mécani...\n",
            "656               N44  ...                          Manoeuvre du réseau ferré\n",
            "\n",
            "[532 rows x 3 columns]\n",
            "    major_code  ...                                 professional_title\n",
            "2            A  ...                     Engins agricoles et forestiers\n",
            "3            A  ...                  Espaces naturels et espaces verts\n",
            "4            A  ...                     Etudes et assistance technique\n",
            "5            A  ...                                         Production\n",
            "6            A  ...                                  Soins aux animaux\n",
            "..         ...  ...                                                ...\n",
            "120          N  ...  Personnel sédentaire du transport maritime et ...\n",
            "121          N  ...         Personnel de conduite du transport routier\n",
            "122          N  ...       Personnel d'encadrement du transport routier\n",
            "123          N  ...          Personnel navigant du transport terrestre\n",
            "124          N  ...  Personnel sédentaire du transport ferroviaire ...\n",
            "\n",
            "[110 rows x 3 columns]\n",
            "    professional_code  ...                                         rome_title\n",
            "125               A11  ...          Conduite d'engins agricoles et forestiers\n",
            "126               A12  ...                            Bûcheronnage et élagage\n",
            "127               A12  ...                     Entretien des espaces naturels\n",
            "128               A12  ...         Aménagement et entretien des espaces verts\n",
            "129               A12  ...                   Protection du patrimoine naturel\n",
            "..                ...  ...                                                ...\n",
            "652               N43  ...                                 Conduite sur rails\n",
            "653               N43  ...                  Contrôle des transports en commun\n",
            "654               N44  ...                        Circulation du réseau ferré\n",
            "655               N44  ...  Exploitation et manoeuvre des remontées mécani...\n",
            "656               N44  ...                          Manoeuvre du réseau ferré\n",
            "\n",
            "[532 rows x 3 columns]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>major_code</th>\n",
              "      <th>major_title</th>\n",
              "      <th>professional_code</th>\n",
              "      <th>professional_title</th>\n",
              "      <th>rome_code</th>\n",
              "      <th>rome_title</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>A</td>\n",
              "      <td>Agriculture et Pêche, Espaces naturels et Espa...</td>\n",
              "      <td>A11</td>\n",
              "      <td>Engins agricoles et forestiers</td>\n",
              "      <td>A1101</td>\n",
              "      <td>Conduite d'engins agricoles et forestiers</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A</td>\n",
              "      <td>Agriculture et Pêche, Espaces naturels et Espa...</td>\n",
              "      <td>A12</td>\n",
              "      <td>Espaces naturels et espaces verts</td>\n",
              "      <td>A1201</td>\n",
              "      <td>Bûcheronnage et élagage</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>A</td>\n",
              "      <td>Agriculture et Pêche, Espaces naturels et Espa...</td>\n",
              "      <td>A12</td>\n",
              "      <td>Espaces naturels et espaces verts</td>\n",
              "      <td>A1202</td>\n",
              "      <td>Entretien des espaces naturels</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>A</td>\n",
              "      <td>Agriculture et Pêche, Espaces naturels et Espa...</td>\n",
              "      <td>A12</td>\n",
              "      <td>Espaces naturels et espaces verts</td>\n",
              "      <td>A1203</td>\n",
              "      <td>Aménagement et entretien des espaces verts</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>A</td>\n",
              "      <td>Agriculture et Pêche, Espaces naturels et Espa...</td>\n",
              "      <td>A12</td>\n",
              "      <td>Espaces naturels et espaces verts</td>\n",
              "      <td>A1204</td>\n",
              "      <td>Protection du patrimoine naturel</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>527</th>\n",
              "      <td>N</td>\n",
              "      <td>Transport et Logistique</td>\n",
              "      <td>N43</td>\n",
              "      <td>Personnel navigant du transport terrestre</td>\n",
              "      <td>N4301</td>\n",
              "      <td>Conduite sur rails</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>528</th>\n",
              "      <td>N</td>\n",
              "      <td>Transport et Logistique</td>\n",
              "      <td>N43</td>\n",
              "      <td>Personnel navigant du transport terrestre</td>\n",
              "      <td>N4302</td>\n",
              "      <td>Contrôle des transports en commun</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>529</th>\n",
              "      <td>N</td>\n",
              "      <td>Transport et Logistique</td>\n",
              "      <td>N44</td>\n",
              "      <td>Personnel sédentaire du transport ferroviaire ...</td>\n",
              "      <td>N4401</td>\n",
              "      <td>Circulation du réseau ferré</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>530</th>\n",
              "      <td>N</td>\n",
              "      <td>Transport et Logistique</td>\n",
              "      <td>N44</td>\n",
              "      <td>Personnel sédentaire du transport ferroviaire ...</td>\n",
              "      <td>N4402</td>\n",
              "      <td>Exploitation et manoeuvre des remontées mécani...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>531</th>\n",
              "      <td>N</td>\n",
              "      <td>Transport et Logistique</td>\n",
              "      <td>N44</td>\n",
              "      <td>Personnel sédentaire du transport ferroviaire ...</td>\n",
              "      <td>N4403</td>\n",
              "      <td>Manoeuvre du réseau ferré</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>532 rows × 6 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "    major_code  ...                                         rome_title\n",
              "0            A  ...          Conduite d'engins agricoles et forestiers\n",
              "1            A  ...                            Bûcheronnage et élagage\n",
              "2            A  ...                     Entretien des espaces naturels\n",
              "3            A  ...         Aménagement et entretien des espaces verts\n",
              "4            A  ...                   Protection du patrimoine naturel\n",
              "..         ...  ...                                                ...\n",
              "527          N  ...                                 Conduite sur rails\n",
              "528          N  ...                  Contrôle des transports en commun\n",
              "529          N  ...                        Circulation du réseau ferré\n",
              "530          N  ...  Exploitation et manoeuvre des remontées mécani...\n",
              "531          N  ...                          Manoeuvre du réseau ferré\n",
              "\n",
              "[532 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W0QqK02UJogF"
      },
      "source": [
        "principal_tree.to_csv(\"principal_tree.csv\", index=False)\n",
        "thematic_tree.to_csv(\"thematic_tree.csv\", index=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7dugeYglHtXR"
      },
      "source": [
        "In the Pole Emploi database, there are many kinds of texts associated to a ROME form. In the `unix_texte_v346_utf8.csv` database, you have three categories:\n",
        "\n",
        "1. `definition`: a description of the job => **used**\n",
        "2. `acces_a_em`: the required experience/diploma \n",
        "3. `cond_exercice_activite`: condition for exercising activity (for example \"The activity may involve handling loads\").\n",
        "\n",
        "For this project, we only used the `definition`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        },
        "id": "R93asmTYgfCZ",
        "outputId": "18779697-2ac2-4ce1-9bbf-9ede106174de"
      },
      "source": [
        "rome_text = pd.read_csv(\"unix_texte_v346_utf8.csv\")\n",
        "rome_def = rome_text[rome_text[\"libelle_type_texte\"]=='definition']\n",
        "rome_def= rome_def.rename(columns={\"libelle_texte\": \"description\",\n",
        "                                   \"code_rome\": \"rome_code\"}).drop(columns = [\"position_libelle_txt\",\"code_type_texte\",\"libelle_type_texte\"])\n",
        "rome_def"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>rome_code</th>\n",
              "      <th>description</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>A1101</td>\n",
              "      <td>Réalise des travaux mécanisés agricoles, sylvi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A1201</td>\n",
              "      <td>Réalise des opérations de coupe et d''entretie...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>A1201</td>\n",
              "      <td>Peut débarder les grumes. Peut estimer la vale...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>A1201</td>\n",
              "      <td>Peut diriger une équipe ou diriger une entrepr...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>A1202</td>\n",
              "      <td>Aménage et entretient des écosystèmes (espaces...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1497</th>\n",
              "      <td>N4402</td>\n",
              "      <td>Peut informer les clients sur les caractéristi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1498</th>\n",
              "      <td>N4402</td>\n",
              "      <td>Peut coordonner l''activité d''une équipe.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1499</th>\n",
              "      <td>N4403</td>\n",
              "      <td>Réalise des opérations de formation de trains ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1500</th>\n",
              "      <td>N4403</td>\n",
              "      <td>Peut acheminer une rame dans un périmètre limité.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1501</th>\n",
              "      <td>N4403</td>\n",
              "      <td>Peut coordonner l''activité de plusieurs salar...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1502 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "     rome_code                                        description\n",
              "0        A1101  Réalise des travaux mécanisés agricoles, sylvi...\n",
              "1        A1201  Réalise des opérations de coupe et d''entretie...\n",
              "2        A1201  Peut débarder les grumes. Peut estimer la vale...\n",
              "3        A1201  Peut diriger une équipe ou diriger une entrepr...\n",
              "4        A1202  Aménage et entretient des écosystèmes (espaces...\n",
              "...        ...                                                ...\n",
              "1497     N4402  Peut informer les clients sur les caractéristi...\n",
              "1498     N4402         Peut coordonner l''activité d''une équipe.\n",
              "1499     N4403  Réalise des opérations de formation de trains ...\n",
              "1500     N4403  Peut acheminer une rame dans un périmètre limité.\n",
              "1501     N4403  Peut coordonner l''activité de plusieurs salar...\n",
              "\n",
              "[1502 rows x 2 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 112
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        },
        "id": "Qf1SW1Xh51fi",
        "outputId": "6afce197-7e1d-4a8a-b4ab-6836b22df0ec"
      },
      "source": [
        "# Concatenation of multiple phrases\n",
        "rome_def = rome_def.groupby(['rome_code'])[\"description\"].apply(lambda x: '\\n'.join(x)).reset_index()\n",
        "rome_def"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>rome_code</th>\n",
              "      <th>description</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>A1101</td>\n",
              "      <td>Réalise des travaux mécanisés agricoles, sylvi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A1201</td>\n",
              "      <td>Réalise des opérations de coupe et d''entretie...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>A1202</td>\n",
              "      <td>Aménage et entretient des écosystèmes (espaces...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>A1203</td>\n",
              "      <td>Aménage et entretient des espaces verts (parcs...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>A1204</td>\n",
              "      <td>Effectue la surveillance de site (espaces natu...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>527</th>\n",
              "      <td>N4301</td>\n",
              "      <td>Conduit un véhicule sur rails (train, métro, ....</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>528</th>\n",
              "      <td>N4302</td>\n",
              "      <td>Contrôle les droits d''accès des voyageurs aux...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>529</th>\n",
              "      <td>N4401</td>\n",
              "      <td>Surveille et coordonne les mouvements des trai...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>530</th>\n",
              "      <td>N4402</td>\n",
              "      <td>Réalise des opérations de mise en fonction, de...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>531</th>\n",
              "      <td>N4403</td>\n",
              "      <td>Réalise des opérations de formation de trains ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>532 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "    rome_code                                        description\n",
              "0       A1101  Réalise des travaux mécanisés agricoles, sylvi...\n",
              "1       A1201  Réalise des opérations de coupe et d''entretie...\n",
              "2       A1202  Aménage et entretient des écosystèmes (espaces...\n",
              "3       A1203  Aménage et entretient des espaces verts (parcs...\n",
              "4       A1204  Effectue la surveillance de site (espaces natu...\n",
              "..        ...                                                ...\n",
              "527     N4301  Conduit un véhicule sur rails (train, métro, ....\n",
              "528     N4302  Contrôle les droits d''accès des voyageurs aux...\n",
              "529     N4401  Surveille et coordonne les mouvements des trai...\n",
              "530     N4402  Réalise des opérations de mise en fonction, de...\n",
              "531     N4403  Réalise des opérations de formation de trains ...\n",
              "\n",
              "[532 rows x 2 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 115
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nXR8SB0kKzMB"
      },
      "source": [
        "#we have the database in csv and we put it on gist\n",
        "rome_def.to_csv(\"rome_description.csv\",index=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "96KFULurob1_"
      },
      "source": [
        "# 1 Import of the data and the packages\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CGVQ7JoCCe-f"
      },
      "source": [
        "### 1.1 Import and installation of packages"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oW9nVzT-yxjT"
      },
      "source": [
        "!pip install \"transformers==2.5.1\"\n",
        "#!pip install -U torchtext==0.8.0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EqgZnEyuPpM5"
      },
      "source": [
        "import torch\n",
        "from tqdm.notebook import tqdm\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler, Dataset\n",
        "from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence\n",
        "\n",
        "from transformers.tokenization_camembert import CamembertTokenizer\n",
        "from transformers import CamembertForSequenceClassification, AdamW, get_linear_schedule_with_warmup\n",
        "\n",
        "from sklearn.metrics import classification_report, confusion_matrix, roc_curve, auc, plot_roc_curve, f1_score\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn import preprocessing,svm\n",
        "from sklearn.multiclass import OneVsRestClassifier\n",
        "from sklearn.svm import LinearSVC\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sn\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "from collections import Counter\n",
        "\n",
        "import nltk\n",
        "from tqdm import tqdm\n",
        "from gensim.models.phrases import Phrases, Phraser\n",
        "from nltk.tokenize import TreebankWordTokenizer, TweetTokenizer\n",
        "from gensim.models import KeyedVectors\n",
        "import random\n",
        "\n",
        "\n",
        "seed_val = 42"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j5WUh2WxEBq_"
      },
      "source": [
        "### 1.2 Import of pretrained word2vec\n",
        "\n",
        "In this project, due to the few number of observations, we used pre-trained word embedding from a French corpus available at https://fauconnier.github.io. The model was trained with the [FrWac corpus](https://wacky.sslmit.unibo.it/doku.php?id=corpora) of  1.6 billion of words, with a skip-gram approach and an embedding size of 200."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RsrH5U4cwqXg"
      },
      "source": [
        "!wget  https://s3.us-east-2.amazonaws.com/embeddings.net/embeddings/frWac_non_lem_no_postag_no_phrase_200_skip_cut100.bin"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xEVYq-asxkyu"
      },
      "source": [
        "w2vecembedding = KeyedVectors.load_word2vec_format(\"frWac_non_lem_no_postag_no_phrase_200_skip_cut100.bin\", binary=True, unicode_errors=\"ignore\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c3YTFMaFFCMK"
      },
      "source": [
        "### 1.3 Downloading the data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AKf1vfiCMs0f"
      },
      "source": [
        "!wget https://gist.github.com/AQLT/329c4af5e9418e94debe0ed721ba71ea/archive/0db14b6a9c9be77cbef12e4b3804e5e279e42aee.zip -O data.zip\n",
        "!unzip -d data/ -j  -o data.zip\n",
        "!rm data.zip"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 480
        },
        "id": "3Abf2rxOpC4f",
        "outputId": "df9de022-a2c0-4277-b4ed-5c8eab68c7ea"
      },
      "source": [
        "principal_tree = pd.read_csv(\"data/principal_tree.csv\")\n",
        "thematic_tree = pd.read_csv(\"data/thematic_tree.csv\")\n",
        "onisep_description = pd.read_csv(\"data/onisep_description.csv\")\n",
        "rome_description = pd.read_csv(\"data/rome_description.csv\")\n",
        "rome_description[\"table\"] = \"rome\"\n",
        "onisep_description[\"table\"] = \"onisep\"\n",
        "onisep_description = onisep_description.drop(columns=[\"metier\",\"GFE\",\"domaine_ONISEP\",\"sous_domaine_ONISEP\"])\n",
        "print(\"Number of ROME descriptions (Pôle Emploi) : \",len(rome_description))\n",
        "print(\"Number of JOB APPELLATION descriptions (Onisep) : \", len(onisep_description))\n",
        "data = pd.concat([rome_description, onisep_description])\n",
        "print(\"Total number of job descriptions : \", len(data))\n",
        "data"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of ROME descriptions (Pôle Emploi) :  532\n",
            "Number of JOB APPELLATION descriptions (Onisep) :  715\n",
            "Total number of job descriptions :  1247\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>rome_code</th>\n",
              "      <th>description</th>\n",
              "      <th>table</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>A1101</td>\n",
              "      <td>Réalise des travaux mécanisés agricoles, sylvi...</td>\n",
              "      <td>rome</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A1201</td>\n",
              "      <td>Réalise des opérations de coupe et d''entretie...</td>\n",
              "      <td>rome</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>A1202</td>\n",
              "      <td>Aménage et entretient des écosystèmes (espaces...</td>\n",
              "      <td>rome</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>A1203</td>\n",
              "      <td>Aménage et entretient des espaces verts (parcs...</td>\n",
              "      <td>rome</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>A1204</td>\n",
              "      <td>Effectue la surveillance de site (espaces natu...</td>\n",
              "      <td>rome</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>710</th>\n",
              "      <td>F1105</td>\n",
              "      <td>Le pédologue est un scientifique spécialiste d...</td>\n",
              "      <td>onisep</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>711</th>\n",
              "      <td>M1403</td>\n",
              "      <td>Prévoir la durée de vie d'un produit industrie...</td>\n",
              "      <td>onisep</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>712</th>\n",
              "      <td>H1206</td>\n",
              "      <td>L'ingénieur en caractérisation des matériaux c...</td>\n",
              "      <td>onisep</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>713</th>\n",
              "      <td>L1301</td>\n",
              "      <td>Quand les 3 coups résonnent dans la salle et q...</td>\n",
              "      <td>onisep</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>714</th>\n",
              "      <td>C1106</td>\n",
              "      <td>Conseiller technique et souvent arbitre, l'exp...</td>\n",
              "      <td>onisep</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1247 rows × 3 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "    rome_code                                        description   table\n",
              "0       A1101  Réalise des travaux mécanisés agricoles, sylvi...    rome\n",
              "1       A1201  Réalise des opérations de coupe et d''entretie...    rome\n",
              "2       A1202  Aménage et entretient des écosystèmes (espaces...    rome\n",
              "3       A1203  Aménage et entretient des espaces verts (parcs...    rome\n",
              "4       A1204  Effectue la surveillance de site (espaces natu...    rome\n",
              "..        ...                                                ...     ...\n",
              "710     F1105  Le pédologue est un scientifique spécialiste d...  onisep\n",
              "711     M1403  Prévoir la durée de vie d'un produit industrie...  onisep\n",
              "712     H1206  L'ingénieur en caractérisation des matériaux c...  onisep\n",
              "713     L1301  Quand les 3 coups résonnent dans la salle et q...  onisep\n",
              "714     C1106  Conseiller technique et souvent arbitre, l'exp...  onisep\n",
              "\n",
              "[1247 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0z_AfhUtyhZB"
      },
      "source": [
        "As described in the section 1, we will focus on the principal tree. We will try to predict the *major_code* (there are 14 major fields) with the description obtained in of 532 ROME forms provided by Pole Emploi and 715 job descriptions provided by Onisep."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 597
        },
        "id": "f8uIdojvt9oq",
        "outputId": "97c14c41-a772-4d35-db47-e65efbf64244"
      },
      "source": [
        "data = pd.merge(data, principal_tree)\n",
        "data = data.drop(columns = [\"professional_code\", \"code_type_referentiel\",\"professional_title\"])\n",
        "data"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>rome_code</th>\n",
              "      <th>description</th>\n",
              "      <th>table</th>\n",
              "      <th>major_code</th>\n",
              "      <th>major_title</th>\n",
              "      <th>rome_title</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>A1101</td>\n",
              "      <td>Réalise des travaux mécanisés agricoles, sylvi...</td>\n",
              "      <td>rome</td>\n",
              "      <td>A</td>\n",
              "      <td>Agriculture et Pêche, Espaces naturels et Espa...</td>\n",
              "      <td>Conduite d'engins agricoles et forestiers</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A1101</td>\n",
              "      <td>Tracteur, moissonneuse-batteuse, ensileuse, ép...</td>\n",
              "      <td>onisep</td>\n",
              "      <td>A</td>\n",
              "      <td>Agriculture et Pêche, Espaces naturels et Espa...</td>\n",
              "      <td>Conduite d'engins agricoles et forestiers</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>A1101</td>\n",
              "      <td>Depuis la cabine de sa machine, le conducteur ...</td>\n",
              "      <td>onisep</td>\n",
              "      <td>A</td>\n",
              "      <td>Agriculture et Pêche, Espaces naturels et Espa...</td>\n",
              "      <td>Conduite d'engins agricoles et forestiers</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>A1201</td>\n",
              "      <td>Réalise des opérations de coupe et d''entretie...</td>\n",
              "      <td>rome</td>\n",
              "      <td>A</td>\n",
              "      <td>Agriculture et Pêche, Espaces naturels et Espa...</td>\n",
              "      <td>Bûcheronnage et élagage</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>A1201</td>\n",
              "      <td>Avec ses cordes de rappel, son harnais, ses mo...</td>\n",
              "      <td>onisep</td>\n",
              "      <td>A</td>\n",
              "      <td>Agriculture et Pêche, Espaces naturels et Espa...</td>\n",
              "      <td>Bûcheronnage et élagage</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1242</th>\n",
              "      <td>N4401</td>\n",
              "      <td>Surveille et coordonne les mouvements des trai...</td>\n",
              "      <td>rome</td>\n",
              "      <td>N</td>\n",
              "      <td>Transport et Logistique</td>\n",
              "      <td>Circulation du réseau ferré</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1243</th>\n",
              "      <td>N4401</td>\n",
              "      <td>Le technicien de la circulation ferroviaire SN...</td>\n",
              "      <td>onisep</td>\n",
              "      <td>N</td>\n",
              "      <td>Transport et Logistique</td>\n",
              "      <td>Circulation du réseau ferré</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1244</th>\n",
              "      <td>N4402</td>\n",
              "      <td>Réalise des opérations de mise en fonction, de...</td>\n",
              "      <td>rome</td>\n",
              "      <td>N</td>\n",
              "      <td>Transport et Logistique</td>\n",
              "      <td>Exploitation et manoeuvre des remontées mécani...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1245</th>\n",
              "      <td>N4402</td>\n",
              "      <td>Sur les pistes de ski, une panne d'une télécab...</td>\n",
              "      <td>onisep</td>\n",
              "      <td>N</td>\n",
              "      <td>Transport et Logistique</td>\n",
              "      <td>Exploitation et manoeuvre des remontées mécani...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1246</th>\n",
              "      <td>N4403</td>\n",
              "      <td>Réalise des opérations de formation de trains ...</td>\n",
              "      <td>rome</td>\n",
              "      <td>N</td>\n",
              "      <td>Transport et Logistique</td>\n",
              "      <td>Manoeuvre du réseau ferré</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1247 rows × 6 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "     rome_code  ...                                         rome_title\n",
              "0        A1101  ...          Conduite d'engins agricoles et forestiers\n",
              "1        A1101  ...          Conduite d'engins agricoles et forestiers\n",
              "2        A1101  ...          Conduite d'engins agricoles et forestiers\n",
              "3        A1201  ...                            Bûcheronnage et élagage\n",
              "4        A1201  ...                            Bûcheronnage et élagage\n",
              "...        ...  ...                                                ...\n",
              "1242     N4401  ...                        Circulation du réseau ferré\n",
              "1243     N4401  ...                        Circulation du réseau ferré\n",
              "1244     N4402  ...  Exploitation et manoeuvre des remontées mécani...\n",
              "1245     N4402  ...  Exploitation et manoeuvre des remontées mécani...\n",
              "1246     N4403  ...                          Manoeuvre du réseau ferré\n",
              "\n",
              "[1247 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NLN0XAx4gN1w"
      },
      "source": [
        "We can notice that the dataset is not perfectly balanced :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yaMnW_zmgKgW",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 959
        },
        "outputId": "88e00fc5-66e6-4059-b074-a44079ec5bff"
      },
      "source": [
        "data[['major_code', 'table',\"rome_code\"]].groupby(['major_code', 'table']).count()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>rome_code</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>major_code</th>\n",
              "      <th>table</th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">A</th>\n",
              "      <th>onisep</th>\n",
              "      <td>44</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rome</th>\n",
              "      <td>30</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">B</th>\n",
              "      <th>onisep</th>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rome</th>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">C</th>\n",
              "      <th>onisep</th>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rome</th>\n",
              "      <td>25</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">D</th>\n",
              "      <th>onisep</th>\n",
              "      <td>35</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rome</th>\n",
              "      <td>39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">E</th>\n",
              "      <th>onisep</th>\n",
              "      <td>43</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rome</th>\n",
              "      <td>23</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">F</th>\n",
              "      <th>onisep</th>\n",
              "      <td>58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rome</th>\n",
              "      <td>38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">G</th>\n",
              "      <th>onisep</th>\n",
              "      <td>34</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rome</th>\n",
              "      <td>30</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">H</th>\n",
              "      <th>onisep</th>\n",
              "      <td>122</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rome</th>\n",
              "      <td>94</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">I</th>\n",
              "      <th>onisep</th>\n",
              "      <td>31</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rome</th>\n",
              "      <td>28</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">J</th>\n",
              "      <th>onisep</th>\n",
              "      <td>38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rome</th>\n",
              "      <td>32</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">K</th>\n",
              "      <th>onisep</th>\n",
              "      <td>132</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rome</th>\n",
              "      <td>71</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">L</th>\n",
              "      <th>onisep</th>\n",
              "      <td>33</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rome</th>\n",
              "      <td>22</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">M</th>\n",
              "      <th>onisep</th>\n",
              "      <td>78</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rome</th>\n",
              "      <td>44</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">N</th>\n",
              "      <th>onisep</th>\n",
              "      <td>27</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>rome</th>\n",
              "      <td>37</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                   rome_code\n",
              "major_code table            \n",
              "A          onisep         44\n",
              "           rome           30\n",
              "B          onisep         21\n",
              "           rome           19\n",
              "C          onisep         19\n",
              "           rome           25\n",
              "D          onisep         35\n",
              "           rome           39\n",
              "E          onisep         43\n",
              "           rome           23\n",
              "F          onisep         58\n",
              "           rome           38\n",
              "G          onisep         34\n",
              "           rome           30\n",
              "H          onisep        122\n",
              "           rome           94\n",
              "I          onisep         31\n",
              "           rome           28\n",
              "J          onisep         38\n",
              "           rome           32\n",
              "K          onisep        132\n",
              "           rome           71\n",
              "L          onisep         33\n",
              "           rome           22\n",
              "M          onisep         78\n",
              "           rome           44\n",
              "N          onisep         27\n",
              "           rome           37"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yFii0Rg1Cxt0"
      },
      "source": [
        "# 2 Model 1 = Baseline: SVM on word2vec sentence-embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WAR0GgkJ-r-Y"
      },
      "source": [
        "To perform Sequence Classification, we used different models.\n",
        "\n",
        "For **Model 1**, we used pre-trained word-embeddings obtained from a French corpus  trained with a **skip-gram word2vec** approach and an embedding of size 200. It is available at https://fauconnier.github.io. The model was trained in the [FrWac corpus](https://wacky.sslmit.unibo.it/doku.php?id=corpora) of  1.6 billion of words.\n",
        "\n",
        "It has the drawback to probably be a bit less performant than if we trained it ourselves in the specific domain of jobs, but we made this choice due to our few number of observations. In this simple model, each **sentence-embedding** is obtained by performing the average of word-embeddings of all words in the sentence. Finally, we use the supervised learning model **SVM** to perform data classification. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        },
        "id": "-xVqrxRV0w85",
        "outputId": "b0a0b685-ab13-4a3e-bb6d-2429467df044"
      },
      "source": [
        "# We keep only the relevant columns : we want to predict the major code (grand domaine)\n",
        "# using the description\n",
        "y_variable = \"major_code\"\n",
        "data_baseline = data[[\"description\", y_variable]].copy()\n",
        "data_baseline"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>description</th>\n",
              "      <th>major_code</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Réalise des travaux mécanisés agricoles, sylvi...</td>\n",
              "      <td>A</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Tracteur, moissonneuse-batteuse, ensileuse, ép...</td>\n",
              "      <td>A</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Depuis la cabine de sa machine, le conducteur ...</td>\n",
              "      <td>A</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Réalise des opérations de coupe et d''entretie...</td>\n",
              "      <td>A</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Avec ses cordes de rappel, son harnais, ses mo...</td>\n",
              "      <td>A</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1242</th>\n",
              "      <td>Surveille et coordonne les mouvements des trai...</td>\n",
              "      <td>N</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1243</th>\n",
              "      <td>Le technicien de la circulation ferroviaire SN...</td>\n",
              "      <td>N</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1244</th>\n",
              "      <td>Réalise des opérations de mise en fonction, de...</td>\n",
              "      <td>N</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1245</th>\n",
              "      <td>Sur les pistes de ski, une panne d'une télécab...</td>\n",
              "      <td>N</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1246</th>\n",
              "      <td>Réalise des opérations de formation de trains ...</td>\n",
              "      <td>N</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1247 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                            description major_code\n",
              "0     Réalise des travaux mécanisés agricoles, sylvi...          A\n",
              "1     Tracteur, moissonneuse-batteuse, ensileuse, ép...          A\n",
              "2     Depuis la cabine de sa machine, le conducteur ...          A\n",
              "3     Réalise des opérations de coupe et d''entretie...          A\n",
              "4     Avec ses cordes de rappel, son harnais, ses mo...          A\n",
              "...                                                 ...        ...\n",
              "1242  Surveille et coordonne les mouvements des trai...          N\n",
              "1243  Le technicien de la circulation ferroviaire SN...          N\n",
              "1244  Réalise des opérations de mise en fonction, de...          N\n",
              "1245  Sur les pistes de ski, une panne d'une télécab...          N\n",
              "1246  Réalise des opérations de formation de trains ...          N\n",
              "\n",
              "[1247 rows x 2 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 56
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3b02b0EI0w86",
        "outputId": "6b51c09c-7f5c-465b-d6ca-a4d2a1ac05ff"
      },
      "source": [
        "# we change the grades to 0-numbering because it is the classical way to do to perform the models\n",
        "tag_to_ix = {}\n",
        "i=0\n",
        "for code in data_baseline[y_variable].unique():\n",
        "  tag_to_ix[code]=i\n",
        "  i += 1\n",
        "print(tag_to_ix)\n",
        "data_baseline['label'] = data_baseline[y_variable].replace(tag_to_ix)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'A': 0, 'B': 1, 'C': 2, 'D': 3, 'E': 4, 'F': 5, 'G': 6, 'H': 7, 'I': 8, 'J': 9, 'K': 10, 'L': 11, 'M': 12, 'N': 13}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n_nDgNbx0w86"
      },
      "source": [
        "# we tokenize using tweetTokenizer because the descriptions are quite short\n",
        "tokenizer = TweetTokenizer()\n",
        "def tokenize (text):\n",
        "  tokens = tokenizer.tokenize(text)\n",
        "  tokens = list(map(lambda x: x.lower(), tokens))\n",
        "  return tokens"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tF4fpcmY0w87"
      },
      "source": [
        "#count  the number of occurences of each word\n",
        "counts = Counter()\n",
        "for index, row in data_baseline.iterrows():\n",
        "    counts.update(tokenize(row['description']))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RiQquGpX3_7m"
      },
      "source": [
        "# Create 3 objects\n",
        "## W : embedding_matrix (1 line = 1 token and columns = embedding dimensions)\n",
        "## vocab : vocabulary\n",
        "## vocab_to_idx : dictionnary containing the link between each token and its id\n",
        "def get_emb_matrix(pretrained, word_counts):\n",
        "    \"\"\" Creates embedding matrix from word vectors\"\"\"\n",
        "    emb_size = pretrained.vector_size\n",
        "    vocab_size = len(word_counts) + 2\n",
        "    vocab_to_idx = {}\n",
        "    vocab = [\"\", \"UNK\"]\n",
        "    W = np.zeros((vocab_size, emb_size), dtype=\"float32\")\n",
        "    W[0] = np.zeros(emb_size, dtype='float32') # adding a vector for padding\n",
        "    W[1] = np.random.uniform(-0.25, 0.25, emb_size) # adding a vector for unknown words \n",
        "    vocab_to_idx[\"UNK\"] = 1\n",
        "    i = 2\n",
        "    for word in word_counts:\n",
        "        if word in pretrained.vocab:\n",
        "            W[i] = pretrained[word]\n",
        "        else:\n",
        "            W[i] = np.random.uniform(-0.25,0.25, emb_size)\n",
        "        vocab_to_idx[word] = i\n",
        "        vocab.append(word)\n",
        "        i += 1   \n",
        "    return W, np.array(vocab), vocab_to_idx"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l5xS3GXW4mxW"
      },
      "source": [
        "pretrained_weights, vocab, vocab2index = get_emb_matrix(w2vecembedding, counts)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "op0xtnn64vAb"
      },
      "source": [
        "# Encode the sentence (sentence embedding = average of word-embeddings)\n",
        "def encode_sentence(text,vocab2index,pretrained_weights, emb_size):\n",
        "    tokenized = tokenize(text)\n",
        "    encoded = np.zeros(emb_size)\n",
        "    for w in tokenized:\n",
        "      encoded+= pretrained_weights[vocab2index.get(w, vocab2index[\"UNK\"])]\n",
        "    encoded = encoded/len(tokenized)\n",
        "    return encoded#.tolist()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 258
        },
        "id": "W75-cVSB0w88",
        "outputId": "c1b4465a-e1c1-497d-c370-0d261f01a21c"
      },
      "source": [
        "# we create a new variable with the sentence embedding\n",
        "data_baseline['encoded'] = data_baseline['description'].apply(lambda x: encode_sentence(x,\n",
        "                                                                                vocab2index,\n",
        "                                                                                pretrained_weights,\n",
        "                                                                                w2vecembedding.vector_size))\n",
        "data_baseline.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>description</th>\n",
              "      <th>major_code</th>\n",
              "      <th>label</th>\n",
              "      <th>encoded</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Réalise des travaux mécanisés agricoles, sylvi...</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "      <td>[0.013002759400898447, -0.043655207058271535, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Tracteur, moissonneuse-batteuse, ensileuse, ép...</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "      <td>[0.018618605980325654, -0.07725054811031652, 0...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Depuis la cabine de sa machine, le conducteur ...</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "      <td>[0.00033290044558868256, -0.09035057332036057,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Réalise des opérations de coupe et d''entretie...</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "      <td>[-0.01938517735917357, -0.047998593846053074, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Avec ses cordes de rappel, son harnais, ses mo...</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "      <td>[0.012706873127116033, -0.0692063626457149, 0....</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                         description  ...                                            encoded\n",
              "0  Réalise des travaux mécanisés agricoles, sylvi...  ...  [0.013002759400898447, -0.043655207058271535, ...\n",
              "1  Tracteur, moissonneuse-batteuse, ensileuse, ép...  ...  [0.018618605980325654, -0.07725054811031652, 0...\n",
              "2  Depuis la cabine de sa machine, le conducteur ...  ...  [0.00033290044558868256, -0.09035057332036057,...\n",
              "3  Réalise des opérations de coupe et d''entretie...  ...  [-0.01938517735917357, -0.047998593846053074, ...\n",
              "4  Avec ses cordes de rappel, son harnais, ses mo...  ...  [0.012706873127116033, -0.0692063626457149, 0....\n",
              "\n",
              "[5 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mmiyed6tQp__",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 959
        },
        "outputId": "10b32d93-8026-4355-c304-9264f894add3"
      },
      "source": [
        "# create train 70 %\n",
        "# create test 30 %\n",
        "X_tr, X_val, y_train, y_valid = train_test_split(data_baseline.index.values, \n",
        "                                                  data_baseline.label.values, \n",
        "                                                  test_size=0.3, \n",
        "                                                  random_state=seed_val, \n",
        "                                                  stratify=data_baseline.label.values)\n",
        "\n",
        "data_baseline['data_type'] = 'not_set'\n",
        "data_baseline.loc[X_tr, 'data_type'] = 'train'\n",
        "data_baseline.loc[X_val, 'data_type'] = 'val'\n",
        "X_train = list(data_baseline[data_baseline.data_type=='train'][\"encoded\"])\n",
        "X_valid = list(data_baseline[data_baseline.data_type=='val'][\"encoded\"])\n",
        "y_valid_desc = list(data_baseline[data_baseline.data_type=='val'][\"description\"])\n",
        "y_train_desc = list(data_baseline[data_baseline.data_type=='train'][\"description\"])\n",
        "y_train = list(data_baseline[data_baseline.data_type=='train'][\"label\"])\n",
        "y_valid = list(data_baseline[data_baseline.data_type=='val'][\"label\"])\n",
        "data_baseline.groupby(['major_code', 'label', 'data_type']).count()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>description</th>\n",
              "      <th>encoded</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>major_code</th>\n",
              "      <th>label</th>\n",
              "      <th>data_type</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">A</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">0</th>\n",
              "      <th>train</th>\n",
              "      <td>52</td>\n",
              "      <td>52</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>22</td>\n",
              "      <td>22</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">B</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">1</th>\n",
              "      <th>train</th>\n",
              "      <td>28</td>\n",
              "      <td>28</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>12</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">C</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">2</th>\n",
              "      <th>train</th>\n",
              "      <td>31</td>\n",
              "      <td>31</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>13</td>\n",
              "      <td>13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">D</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
              "      <th>train</th>\n",
              "      <td>52</td>\n",
              "      <td>52</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>22</td>\n",
              "      <td>22</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">E</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">4</th>\n",
              "      <th>train</th>\n",
              "      <td>46</td>\n",
              "      <td>46</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>20</td>\n",
              "      <td>20</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">F</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">5</th>\n",
              "      <th>train</th>\n",
              "      <td>67</td>\n",
              "      <td>67</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>29</td>\n",
              "      <td>29</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">G</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">6</th>\n",
              "      <th>train</th>\n",
              "      <td>45</td>\n",
              "      <td>45</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>19</td>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">H</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">7</th>\n",
              "      <th>train</th>\n",
              "      <td>151</td>\n",
              "      <td>151</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>65</td>\n",
              "      <td>65</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">I</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">8</th>\n",
              "      <th>train</th>\n",
              "      <td>41</td>\n",
              "      <td>41</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>18</td>\n",
              "      <td>18</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">J</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">9</th>\n",
              "      <th>train</th>\n",
              "      <td>49</td>\n",
              "      <td>49</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>21</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">K</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">10</th>\n",
              "      <th>train</th>\n",
              "      <td>142</td>\n",
              "      <td>142</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>61</td>\n",
              "      <td>61</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">L</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">11</th>\n",
              "      <th>train</th>\n",
              "      <td>38</td>\n",
              "      <td>38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>17</td>\n",
              "      <td>17</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">M</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">12</th>\n",
              "      <th>train</th>\n",
              "      <td>85</td>\n",
              "      <td>85</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>37</td>\n",
              "      <td>37</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">N</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">13</th>\n",
              "      <th>train</th>\n",
              "      <td>45</td>\n",
              "      <td>45</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>19</td>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                            description  encoded\n",
              "major_code label data_type                      \n",
              "A          0     train               52       52\n",
              "                 val                 22       22\n",
              "B          1     train               28       28\n",
              "                 val                 12       12\n",
              "C          2     train               31       31\n",
              "                 val                 13       13\n",
              "D          3     train               52       52\n",
              "                 val                 22       22\n",
              "E          4     train               46       46\n",
              "                 val                 20       20\n",
              "F          5     train               67       67\n",
              "                 val                 29       29\n",
              "G          6     train               45       45\n",
              "                 val                 19       19\n",
              "H          7     train              151      151\n",
              "                 val                 65       65\n",
              "I          8     train               41       41\n",
              "                 val                 18       18\n",
              "J          9     train               49       49\n",
              "                 val                 21       21\n",
              "K          10    train              142      142\n",
              "                 val                 61       61\n",
              "L          11    train               38       38\n",
              "                 val                 17       17\n",
              "M          12    train               85       85\n",
              "                 val                 37       37\n",
              "N          13    train               45       45\n",
              "                 val                 19       19"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ubLvNI287T0i",
        "outputId": "b5fabf21-345d-463f-8c52-6325724b18be"
      },
      "source": [
        "# we reshape the data as a big array\n",
        "X_train_cv = np.vstack(np.array(X_train))\n",
        "print(X_train_cv.shape)\n",
        "X_val_cv = np.vstack(np.array(X_valid))\n",
        "X_val_cv.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(872, 200)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(375, 200)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 87
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lceYgstu7T0i"
      },
      "source": [
        "We do a binarization of labels : instead of a number from 0 to 13 we will have a dummy variable of the number associated to a label."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YTht8lWu7T0j",
        "outputId": "2af6668d-1cd2-492c-ff12-ab9d5bbc004f"
      },
      "source": [
        "lb = preprocessing.LabelBinarizer()\n",
        "lb_trained = lb.fit(y_train)\n",
        "y_train_lb = lb_trained.transform(y_train)\n",
        "y_valid_lb = lb_trained.transform(y_valid)\n",
        "print(y_train_lb.shape, y_valid_lb.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(872, 14) (375, 14)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xznRdfDM7T0j"
      },
      "source": [
        "# we define a SVM model\n",
        "svc = LinearSVC(verbose=1, random_state=42)\n",
        "clf_svm = OneVsRestClassifier(svc, n_jobs=-1 ).fit(X_train_cv, y_train_lb)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MXbVxa5q7T0j",
        "outputId": "19205d3c-3da5-4400-901e-d7c8dce6a32f"
      },
      "source": [
        "# We get the prediction\n",
        "y_pred_svm = clf_svm.predict(X_val_cv)\n",
        "print(y_pred_svm.shape)\n",
        "y_score_svm = clf_svm.decision_function(X_val_cv)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(375, 14)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IQkqXSMRWJfG"
      },
      "source": [
        "# List to go from the number associated to a label to the label\n",
        "key_list = list(tag_to_ix.keys())\n",
        "val_list = list(tag_to_ix.values())\n",
        "def matrix_pred_baseline (y_true, y_pred, lb_trained):\n",
        "  y_true_names = lb_trained.inverse_transform(y_true)\n",
        "  y_pred_names = lb_trained.inverse_transform(y_pred)\n",
        "  y_pred_names = [key_list[val_list.index(i)] for i in y_pred_names]\n",
        "  y_true_names = [key_list[val_list.index(i)] for i in y_true_names]\n",
        "  data={\"description\":y_valid_desc,\"label\":y_true_names,\"y_pred\":y_pred_names}\n",
        "  return pd.DataFrame(data)\n",
        "def show_eval(y_true, y_pred, lb_trained):\n",
        "  ''' \n",
        "  Show eval metrics.  Takes binarized y true and pred along with trained binarizer for label names\n",
        "  '''\n",
        "  mat = matrix_pred_baseline(y_true, y_pred, lb_trained)\n",
        "  y_pred_names = mat.y_pred\n",
        "  y_true_names = mat.label\n",
        "  print( classification_report(y_true_names, y_pred_names) )\n",
        "  labels = list(tag_to_ix.keys())\n",
        "  cm = confusion_matrix(y_true_names, y_pred_names , labels = key_list) \n",
        "  \n",
        "  df_cm = pd.DataFrame(cm, index=labels, columns=labels)\n",
        "  # config plot sizes\n",
        "  sn.set(font_scale=1.2)\n",
        "  sn.heatmap(df_cm, annot=True, annot_kws={\"size\": 18}, cmap='coolwarm', linewidth=0.5, fmt=\"\")\n",
        "  plt.tick_params(axis='both', which='major', labelsize=10, labelbottom = False, bottom=False, top = False, labeltop=True)\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rrnpAOqSx5YF"
      },
      "source": [
        "To evaluate the different models **quantitatively**, we used **F1-scores** per class and their global weighted and macro values. Indeed, it is more suitable than the accuracy when the dataset is not balanced (like for us) and to account for the weakness of our models in the classification of some specific (and rare) classes. \n",
        "\n",
        "Below are the results of quantitative evaluation of Model 1 (baseline). The weighted average of F1 scores is quite high for a baseline (0.55). We will try to improve the performances using other models LSTM (section 3) and CamemBERT (section 4). "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 647
        },
        "id": "4aXDgp1LWJfK",
        "outputId": "28e301c8-0e95-4ab2-efa5-85cce8251bfd"
      },
      "source": [
        "pred_vs_true_m1 = matrix_pred_baseline(y_valid_lb, y_pred_svm, lb_trained)\n",
        "show_eval(y_valid_lb, y_pred_svm, lb_trained)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           A       0.11      1.00      0.19        22\n",
            "           B       1.00      0.17      0.29        12\n",
            "           C       1.00      0.15      0.27        13\n",
            "           D       1.00      0.36      0.53        22\n",
            "           E       0.75      0.15      0.25        20\n",
            "           F       0.92      0.38      0.54        29\n",
            "           G       1.00      0.37      0.54        19\n",
            "           H       0.82      0.57      0.67        65\n",
            "           I       0.75      0.17      0.27        18\n",
            "           J       0.89      0.81      0.85        21\n",
            "           K       0.97      0.54      0.69        61\n",
            "           L       1.00      0.59      0.74        17\n",
            "           M       0.91      0.27      0.42        37\n",
            "           N       1.00      0.63      0.77        19\n",
            "\n",
            "    accuracy                           0.47       375\n",
            "   macro avg       0.87      0.44      0.50       375\n",
            "weighted avg       0.87      0.47      0.55       375\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEDCAYAAADgNmW7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd1gU1/7G3210lg5KMaIiiIhiA3uNXSyJUawxam5iSdN7c5N4b6K58ZpoTLwxloiJoCQo9ogosVdQiVholtgQQWCBpSwsW35/7A903cLM7uzuYZnP88zzyMyZd74zc/zumTPnvMNRKpVKsLCwsLBYFK6lA2BhYWFhYZMxCwsLCxGwyZiFhYWFANhkzMLCwkIAbDJmYWFhIQA2GbOwsLAQAFHJ+Pjx4wgODsa9e/eM0unUqRMmTJiA6OhoTJo0CX/++adResXFxfjwww8xfPhwTJ48GQsWLMD9+/cNjmvs2LGIjo7Gzz//DIVCYXBcDXoNy08//cSYVn5+vsFaJSUlWLp0KYYNG4bJkydj6tSp+OOPP2jrREREqP29b98+rFy50uC4dOlaWu/F/c+cOYORI0fiyZMnFoungeDgYCxbtqzxb5lMhqioKPztb38zSGv16tWNf2/btg0//PADI3FaC3xLB/Aihw8fRo8ePZCcnIz33nvPYB07OzscPHgQAHDu3DmsW7cOO3fuNEhLqVRi8eLFmDhxIr777jsAQG5uLkpLSxEYGGhwXKWlpVi6dCmqqqoMPtcX9YyFKS2lUolFixZh4sSJ+PbbbwEAT548wcmTJ43WtnYuXbqE//znP9i2bRv8/PwsHQ4cHBxw584d1NbWws7ODhcuXICPj49BWjY2NkhNTcXbb78Nd3d3hiO1DohpGVdXVyMjIwNfffUVkpOTGdOtqqqCUCg0eP+0tDTw+XzExMQ0rgsJCUHPnj2NisvDwwNffvklEhISYE3zbtLS0iAQCNSul5+fH2bNmmXBqMjnypUrWL58OTZv3ow2bdpYOpxGBg0ahNOnTwMAkpOTMXbsWIN0+Hw+pk6diri4OAajsy6IaRmfOHECAwYMQGBgINzc3HDr1i2EhYUZpFVbW4sJEyagrq4OxcXFRlWAO3fuoHPnzgbvr4+AgADI5XKUlpbC09OT9v4N59nA3/72N4wZM8agWF7U8vf3x48//miQzp07dxAaGmrQvvpiAoCKigoMHTqUEW2SkEqlWLRoEeLj49G+fXtLh6PGmDFjsHHjRgwZMgR5eXl47bXXkJGRYZDWjBkzEB0djfnz5zMcpXVATDJOTk7G7NmzAagqQHJyssHJ+MVH7mvXruHjjz/G4cOHweFwGIuXBEjspniZFStWICMjAwKBAHv37jUqpn379uHWrVtMh2hx+Hw+IiIisGfPHixfvtzS4agREhKC/Px8HD58GIMGDTJKy8nJCRMmTEB8fDzs7OwYitB6IKKbory8HGlpaVi+fDmGDh2Kbdu2ISUlhZHH94iICJSVlUEkEhm0f1BQELKysoyOQxuPHz8Gj8eDh4eHSfQtQVBQELKzsxv//vzzz7F9+3aUlZVZMCqy4XK5+P7773Hz5k1s3rzZ0uFoMHToUHzzzTcGd1G8yJw5c7B3715IJBIGIrMuiEjGx44dw4QJE3Dq1CmcPHkSZ86cgb+/P65evWq09r179yCXy+Hq6mrQ/lFRUZBKpdi1a1fjutzcXKNjE4lE+PzzzzFjxgyrarFHRUWhrq4Ov/76a+O62tpaC0bUPLC3t8eWLVvw+++/IykpydLhqPH6669j0aJFCA4ONlrL1dUVo0aNwp49exiIzLogopvi8OHDWLBggdq6ESNG4PDhw+jVqxdtvRf7GpVKJb7++mvweDyDYuNwONiwYQNWrVqFrVu3wtbWFn5+fvj0008Njksmk4HH42HChAmYO3euQXG9qNfAgAED1IYiWQIOh4Mff/wR//3vfxEbGwt3d3fY29tbPC5TIZPJYGNjw4iWq6srYmNjMWPGDLi7u2PYsGEG6UgkEgwcOLDx77lz5xpVz1q1atXYhcgEb731FhISEhjTsxY4rIUmC4vh5ObmYvny5WxLj8VoiGgZs7A0R3777Tfs2LHDoKckFpaXYVvGLCwsLARAxAs8FhYWlpYOm4xZWFhYCIBNxiwsLCwEwCZjFhYWFgJgkzELCwsLAbDJmIWFhYUA2GTMwsLCQgBsMmZhYWEhALPNwLt//z4OHTqECxcu4NGjR6irq0ObNm0watQozJkzBw4ODgBUXhKHDh3C6dOncevWLTx79gxubm4ICQnBu+++i65du5orZBYWlmZMsoC6sdHY+jwTRkINs83AW7t2LRJ2xGNA9zB0CWoLPo+Hq9l3cDztGoLa+OLnlR/BzsYGddJ69H9zKTq+4of+EZ3h6+WBknIx9p04j+IyMb54dyamva/yfO0//ozGcd5f0B5Tov1x5mIx0jJEeCXAEa+P88X1rAp88K8bePlsz/8+yOq1GvRMrWXoeZKqBZB3L5tjvTBUy1iaWzI2W8t45MiRmB7ZAU4O9o3rXhveH21aeeHnA6k4dCoNb4wcCB6Pi83/WoIenYLU9p84tA+m/v2/WL/zAN5Y8im4XM0elsA2DnhtnB9OXyzG8v8+99R9WiTBh38LwvCB3vjjzDNK8bJa9LRIjo3Vsg4tunAEzcua1mx9xl26dFFLxA28GtUdAHAv/ykAgM/jaSRiAPBwEaJ7pw4QiStRWlqq9RjDB3qDy+Vg90H1L+v+fuwpJLVyjBjsTTleVoueFsmxsVrWoUUXnj2P8kICFndtKxKVAwDcXZybLPtMVA4Bn6fzA6MhQc6Qy5XIuS1WWy+tV+LOX1UICWr6GKyWYVokx8ZqWYcWXbh8K28ZX716FStWrGDk4HKFAtv2HwWPx8XIvj30lr1wLQtZ9x7i1ajusLW11VrG090GFeJ61Ms0O6FKRHVwc7EBn+INYrXoaZEcG6tlHVp04Qg4lBcSoNQyzs7Oxu+//45jx47Bz88PI0aMYOTg6+L34eadB1g4dRza+vroLPfo6TN8vmkHvN1d8MHMiTrL2dnyUC9TaN0mlSr+vwwXVTJ5k7GxWvS0SI6N1bIOLbo0t5axzmR8//59JCcn4/Dhw3Bzc8OYMWOgVCqxY8cORg68aXcydqeexaShfTF3gu7k/uRZKRau+hEAB+v/8S7chLofa2rr5HCz0/4JHBsb7v+X0V4xWC3jtEiOjdWyDi26kNLipYrOborRo0cjLS0NW7ZswW+//YZZs2ZpHcFgCD/tOYKfDxzD+EGR+GTeVJ3lCopL8e5/foCktg4/froQHdr46tUtEUnhIhRAoOUX0dPdFmUVUsi0PC6xWsZrkRwbq2UdWnTh8jmUFzps3LgRw4cPR48ePRAZGYl58+YhJyencXtwcDDCw8MRERHRuOTlNT10Tmd23bBhA7y8vDB79mwsX74cly5dAhNDkn/acwRb9x3F2IG9sXxBjM4vIxcUl+KdL39AVY0EGz5ZiOC2AU1q596pBI/HQaeO6i/4bAQcBLVzQt7dKspxslr0tEiOjdWyDi268Gy4lBc6jB49Gnv37kVGRgbOnTuHfv36YcGCBVAonrfwt27dimvXrjUuVL6srTOK4cOH47vvvkNKSgoiIyMRFxfX+Hn58+fP0wq+McB9Kdi67yjG9O+Ff789XWdL+2mxCO/+5wdU1kjwwycL0aldG0r6J84VQ6FQ4o0Jfmrrx49sDXs7HlJPF1GOldWip0VybKyWdWjRhcPlUF7oEBgYCBcXl8a/uVwuiouLUVlZaVS8Tb7Ac3BwwPjx4zF+/HhUVFTg6NGj2Lp1K/r370/rQAkJCfhpTwpaebqhd1gwjl7MUNvu4eKMyC4hqJbU4p3//ICCYhGmjhyIh0+f4eFT9UHhw1/pAU9PT41j/PWwGvuSC/D6eD989UkoLmWI0NbfAa+P98O1m+W0BpezWvS0SI6N1bIOLbpweNRbvGKxGGKxWGO9UCjUOpT29OnTWLZsGSorK8HhcDB37ly1BL106VLU19fD19cXMTExeOONN5qMgdY4YxcXF0ydOhVTp+ru59XFzZs3AQCFJWX4YvNOje3dO3VAZJcQVFRVo6BYNalj17GzWrUCo17VmowB4H+xd1H4rBbRI1ujTy8PVIjrsedwAbYl3Nc67VIfrBY9LZJjY7WsQ4sOXB71Fm9cXBw2bNigsX7x4sVYsmSJxvrBgwfj6tWrKC8vx4EDB9C6devGbdu3b0dERAS4XC7S0tKwbNkyyGQyTJ8+XW8MZv06tDjjGCM6wh4jAWifn08XfXP9rUWrQY/VoqcFkHcvW1K9MJb0PpGUy3Y69getlvGLKBQK9OrVC4mJiQgK0pw9vHHjRpw9exaJiYl6dSw+A4+FhYXFFNB5MUcl6epCoVBAJpPh4cOHWpMxl8ulNPjBrC1jFhYWFnORMaQf5bI9Tl2gXDY+Ph6jR4+Gl5cXRCIRvvvuOxw9ehQpKSkoKiqCQqFAcHAwuFwu0tPT8dFHH2HRokWYPXu2Xl0iWsYlJSX43//+hzNnzqC0tBSenp4YPnw43nvvPYN/rVhYWFo2dEdJUKVh/kV1dTWcnJzQpUsXbN++HZ6enrhx4wbWrFmDwsJC8Hg8+Pr64oMPPkBMTEzT8ZqzZbx2n+ZMG0lVKQ78+AZqKosR0vsNuPsEQVR0B7mXd8PNuwOi30kA30bd7W3ZZNXjx9j5t9TWv/maD8YOcUd6ZiXy/qqBTK5EeIgTBvZywV+PJVi66i9I69VPNzk2DAB5XrPN0bfW0PMcNu2yhs6JxN4QV8pw5341gto5okYix4wl17Ue88V9WD9j8uqFpfyMM0cMoFy2W+o5o49nLBZvGWee2oKq8gIMmboWHbqNbVzv0yYCp3Ytw43z29F96LuUtM5frcDuI8WokTxP+ilnylBQVIdp47wxor8bDp8SUdIi1dOVVC2m9Wa+dx1Pn9UBAGLXhMHeznCbQ1KvGatlYj9jE7WMTYXFv4FX8Ndl8AR2aN91jNr69uGjwePb4nbGfspadx/WqiXiBs5eqQAAvOJnR1mLVE9XUrWY1mtIxExA6jVjtUzrZ8zhcikvJGDxlrFcLgWfb6sxLZrD5YIvsEOl6DFqq8tg5+hm8DE83QQAgHKxjPI+pHq6kqplCj2mIPWasVqmrRM8ARlJlioWj9bNuwPqJBUoLchRW19akIM6iapFW1VeYLA+lwPEjPOGTKbE6fRyyvuR6ulKqpYp9JiC1GvGapnYz9hE06FNBe1kLBKJGDEMaiCs32xwOFwc//VDPMo9g6ryAjzOO4sTv30ELk/VopXV1xqs//a01ujUwQE7DxbhSZGU8n5UfVhZLdPpMQWp14zVMm2daG7dFHqjyMzMxKxZs7B48WJkZ2dj3LhxGD9+PPr27YuzZ7VPVaZL68CeGBrzLeql1TgW9w5++3oYjsUvROt2kWgTonqjKrB1Mkh75gRvjB/mgZQzIiSllNDat7ZODgFf++UxxNPV2rVMoccUpF4zVsvEfsbNrGWst8945cqV+Oijj1BZWYk5c+Zg69at6NatG+7du4elS5di4MCBjATRrssotO38KsoKb0NaVw1Xr0DYO3ngwI9vgMPlw8WDmmvbi0yP9kbMeG+kni/Dhh30uzlKRFK0DXCEgM/ReMQyxNPV2rVMoccUpF4zVsu0dYKUJEsVvS1juVyO/v37Y/To0fD09ES3bt0AAO3bt2c+EC4PHr6d0DqwJ+ydPFBTWYySghy0DuypMc64KaZHe2NGtDeOXyjD/+KeNL2DFkj1dCVVyxR6TEHqNWO1TFsnuHwe5YUE9CbjF/2G7ezUh4XpMoVnAqVCgYu/r4JSKUfEkHdo7Rszzgszor1x4mIZvt/+xGBXKFI9XUnVMoUeU5B6zVgt09YJLo9DeSEBvd0Uubm56N69O5RKJerq6tC9e3cAgFKphFRK/WWYPurrqnFg41S0DR0OZ3c/SGurcO96MkqeZKHniA/g256689LYIe6YOdEHz0qlyMypxuBIF7XtZWIZMrOrKWmR6ulKqhbTesMHeMDHU/UVcFdnAfh8DmZMUn12q6ikDsfPlVokLlarGfkZN7NuCr3J+MXvOpkKLk8Aj1bBuHv9MCSVxeAL7OHpH4ZRc7cioCM9A/uObVXdGd4eNlg6z19j+428amRm36esR6qnK6laTOqNHuKFbqHqj7ZvTVXd08xsMa1kzGRcrJbl6xhVSBklQRWLT/rg8W0wNOZbRrS+++UJvvvFsD5ibSgUQOKBfCQeyGe1zKy3dGUuI/E0QOo1Y7VMh1W1jFlYWFiaK80tGbN+xiwsLFbJ44WvUS4bsHGvCSOhhllbxm9/vAWFj7JQ+CgLFSX5ELr7YeGqk1rL3s++gLxrx1D4KAvFT25DLpMi5sN4vBIciblDVGW02fVxOMCUaD9MGOWLVt52KK+Q4tT5YsQmPNA6uLzBqu9lO05fHxsMjXJFRKgTWnvbQCDgoPCZFOczxDjwRwnqpJq/YfrsOA2NiwmtBj1Taxl6nqRqAeTdy+ZYLwzVMha2z1gPZw6sg52jK1oFhKKuRv9nrbMu/47sK4fh5RsEj9bt8ewxtZeJ781/7p2auP+xyjt1vB+C2jnp9E7Vxoj+bo3eyKfTyxu9kWdP8kH/nkKt3sjmiItkLZJjY7WsQ4sWJhx+awrMmozf+fI4XL0CAACxK8dBWlujs+ygCR9i1IyV4AtskJ66jVIyZtI7lfVGtqyfMavFahlLc+szNms7viERU8HZzQd8gQ0tfSa9U1lvZMv6GbNarJaxWJVR0MOHD5GRkaGxPiMjA48ePTJZUIZiDu9U1hu5+cXGalmHFl2am1GQ3mS8atUqODlpOqY5OTlh1apVJgvKUEztncp6IzfP2Fgt69Cii1V5U5SUlCA4OFhjfXBwMJ48YW5yBVOY2juV9UZunrGxWtahRRdTtYw3btyI4cOHo0ePHoiMjMS8efPUZitnZ2dj2rRp6Nq1KwYPHoz4+HhKunqvQmWl7hEPtbWGG76bClN6p7LeyM03NlbLOrRow+VSX2gwevRo7N27FxkZGTh37hz69euHBQsWQKFQoKqqCvPnz0f//v1x+fJlfP/999iwYQOOHj3adLj6NoaFhWH37t0a65OSktC5c2daJ2AOSkRSuAgFEGh57DHGO5UJb2Sm4iJVi+TYWC3r0KILh8OhvIjFYuTn52ssYrFYQzcwMBAuLs9NyLhcLoqLi1FZWYnU1FRwuVwsXLgQtra26NatG6ZMmYJff/21yXj1JuNPP/0U+/btw6xZs7B69WqsXr0aM2fOxJ49e/DZZ58ZcHlMiym8U1lvZOqQGhurZR1adKEzmiIuLg7Dhg3TWOLi4rRqnz59Gj179kSXLl2wevVqzJ07Fy4uLsjNzUVoaKia/XBYWBhyc5v2WtGbjD09PZGYmIhFixbBz88Pfn5+WLRoEXbt2gUvLy+al8b0MO2dynojW87PmNVitYyFw+dRXubMmYMTJ05oLHPmzNGqPXjwYFy9ehXp6en45z//iYiICABAVVUVnJ3VR4gIhUJUVTX9o0Np0kdUVBSioqKoFNXLrbQDqBCpHvNrKkWQy+tx4chGAICLuy/CoiY2ln2Wn4s7N1RTpZ/c+xMAkJV+EPn3MiDJAWbNmqWhz6R3KuuNbFk/Y1aL1TIWOi/mhEIhhEJh0wVfwtXVFbNnz0avXr3Qrl07ODk5obRU3d5VLBZrHZX2MmadgXf9wl48vnNZbd25Q+sBAAFBvdWSceGj7MZtDdy4qDLzOAcgOjpa6zGY8k5lvZEt62fMarFaxsLhmGcyh0KhgEwmw8OHDxESEoKUlBQoFIrGroqsrCyEhIQ0qWPWZDxj6Q7KZcP7TkZ438latzUYBQH3NLYx5Z3KeiNbXo/VYrWMwkSTOeLj4zF69Gh4eXlBJBLhu+++g42NDbp16wY7OzusXbsWmzZtwoIFC5Cbm4ukpCR88cUXTeqyFposLCxWSfnXiymXdf14A+WyCxcuxPXr11FdXQ0nJyd06dIFixcvbhxhlp2djRUrViAnJwdubm6YN28eZs+e3aSuWZPxli1bkJWVhaysLOTn58PPzw8nT2q30HyZNWvWIDY2Fg4ODrh27ZqJI2VhYWnuVKxZQrmsy99/MGEk1DBrN8W6devg4OSCgMBOcCgTQyJVIvV60zPZ8h/k4udftsPWzgEy+fPfDm3eqe8veG7Xl5YhUtn1jfPF9awKrXZ9+vxhDdVamaDpW/Hmqzz4ewLX7yvxpEQJAQ8Ia8uFnycHF7IUOJGpPvD93zP4jMXVEJuptQy9ZqRqAeTWseZULwzVMhYOj4xpzlQxazL+/Icj8PRRObetWjoJdXosNBtQKOT4bcsXCO3WH7WSKjy6l6WzLKnWf/6eQBtvDtJyFEj983nSvXJHjkXjeOgRxMGJTEpSxJ4jybGxWtahRRtC3NioYtZoGxIxHU4fSUBh/l+Y8tYnTZYl1frPVqB6kVApUW8CKBRATR0gpW4AR+w5khwbq2UdWnShMwOPBIj+IKmouADJuzZg9OvvwN3Lt8nypFr/PSlVQlKnRN9QLsqrFapuCj7QtR0Xrd2B5MvU5+aTeo4kx8ZqWYcWbdiWMXPsiv0Snj7+GDKu6TeRALnWf7VSYNcZOSRSYMoAHj6YxMei8Xz06shB0jkFrt2j/g6V1HMkOTZWyzq06GJVfsYvIhKJIBJR+8wQE1w9fwQ5mRcwdcG/weNRa8CTbP0nlQHF5UpczFZg1xk5DqXJIaoEJvfjol0r6pWB5HMkNTZWyzq0aMPhUl8IQG8USqUSP/zwAyIjIzFq1CiMHDkSUVFR2LCB+pg8Q6iuqsC+uK8RNWQy2gV3o7wfqdZ/3q7A3BE8/FWoxPFrCuTlK5F5T4lfUuWokgDjIrmUv51I6jmSHBurZR1adOHweJQXEtCbjLdv344///wTe/bsweXLl3HlyhUkJSXh2rVr2L59u8mCSknahLpaCfoOfw3FhY8al3ppHQDV56CePn2qsR+p1n+RIVwI+BxkP1QvL5MDdwqUcHXiwNWRkhSx50hybKyWdWjRhsuhvhCA3mR88OBBfPvttwgIeD4KIiAgAGvWrMGBAwdMFpSouADSOgm+/XQ6Vr43tnF5ePcmpHUSjBgxAgsWLNDYj1TrP6HK5kJr67ehHlCtD6SeI8mxsVrWoUUXq/ogqUwmg7u7u8Z6d3d3yGQ0xmPR5NWJb+Gtj77VWFr5t4dAYIv169fjk080h7qRav1XrPqgNLq1V8+4tgIg2J8DSZ0SIop1ktRzJDk2Vss6tGjD4VBfCEDvmzGBQGDQNl1cPvs7RMUqC80qsQgyWT2O7t0CAHD38kXvgeMBAIEdtfcTnz36G0TFBRg1atT/r1Gf7UOq9V96ngLh7TgY1o0Lb1clHhcrYW8DRHTgwtmBgyOX5ZTdq0g9R5JjY7WsQ4s2hLR4qaI3Gefm5qJ79+4a65VKJaRS6h/kbODSyX24m31VbV3yLtXLwA6hPRuTsTGQaP1XUQ1sOyrHwC5cBLbioPMrHMjkQGEZ8MefcuQ+phcYiedIemyslnVo0YGUF3NU0ZuMX/ziKRO8/8UvJt+fVOu/sirg4CVm3hqTeo5M67FarJZREDJkjSpEz8BjYWFhMRhCRklQhfUzZmFhsUpqf/uaclm7mI9NGAk1zNoy/td/vseDe3m4fy8XxUUF8PRuje9j9Q+RO3/qCE6k7Mfjh3ehVCrh5d0akyeOw6JFi7Ta9XE4wJRoP0wY5YtW3nYor5Di1PlixCY80Dq4XJ8lIQlar7//l9p6X28Bpox0RaC/Ldxc+ODzgJIyGf7MrsHBkxUoF8u1XUbsWd+OkbgaYtNlVWnIeQ6bdlljfcyE1ggKdERQoAN8fexQWFyHGUuuaz1mAycSezMaF0BuvTD1vSRBy2iaWcvYrMl4945NcHIWom27ENRUNz2W66f1X+LcqSPo1WcI+g0eBQ6Xg+KipygoKNC5z3vzn3unJu5/rPJOHe+HoHZOOr1Tm5OWhysPrkI+Lt+sRmm5HHK5Em18bTC8rxD9ujth2Tf5EFdR75tm8hyZ1JsfEwBxpQx37lfDydH4akrivWS1DKtjlGH7jHWz7qd98G6lGm/4z8UxqK2V6Cx7OvUQzp44jHc+/Bz9h4xR29Yr2FXrPqT6sDKpdfN2LW7e1px9mHOvFkvn+mBIb2ccPFlh9riY1pv53nU8faaacRm7Jgz2doa/GSf1XrJaJvYzbmajKcz609GQiJtCqVTi0J44tG0f3JiIJTXVaKp7m1QfVnN4uhaLVJNwHB2o31KS/YwbEjETkHovWS3T+hk3N6MgIkdTPH3yEM8K8/Hq2CnYn7gNx35PRFWlGPYOjugzcARCV/0Ljo6aZg6k+rCawtNVwOfAzpYDAZ+DgFY2mBmtmil5LVv304ap47Kod62Z4mK1LF/3KWOiPuM1a9bg9OnTePr0KRwcHDBw4ED8/e9/h5ubW2OZ4OBg2NragvdC6zwxMRHBwcE6dclMxvkPAQDp549DJqvHhDfmwsvHF9eunMfJo/vxjugJ4uPjNfZryjs1PNQFfD6HkjEJqVoNDOvjjPmvezb+XVRaj/Xxz5DzVy1lDabjMsV5MgGp95LVMnGdMFGLl8fjYc2aNQgKCoJYLMbf//53fPLJJ9i8ebNaua1btyIyMpKyrt5ot27d2vjvlJQUtW3r1q2jfBC6SCSqb+OJK8rw3sf/xdhJM9G771D87f1/Y8DQsbh8+TLOnj2rsR+pPqym8HS9fKMaK358iq9jC7H7aBlqJAo4O9HTINnPmElIvZeslqn9jE3jTfHRRx8hNDQUAoEAHh4emDVrFi5f1hwRRBe9V+HIkSON//7pp5/Utp07d87og+vCxtYWAODm4YWwbr3Vtg0YqupD1nbypPqwmsLTVVQhx83bEly5WYPdKWXYkFCMWdEemDRc+8tNc8RlSe9afZB6L1ktE9cJLpfyIhaLkZ+fr7GIxeImD3Pp0iWEhIRorF+6dCkiIyMxadIk7N69u0kdvd0UL74we/nlmSnnirh7qDr1Xd08NLa5uqsezbVdpNP2fvgAACAASURBVBKRFG0DHCHgczQeiwzxYSVRSxcPC6S4n1+Hkf2F2H+83CJxmeM8DYHUe8lqmdrPmPpoiri4OK0fzVi8eDGWLFmic78jR44gKSkJO3fuVFu/fft2REREgMvlIi0tDcuWLYNMJsP06dN1h6svwBe/mvryF1RN+UXVgLYdILCxhai0WGObqEQ1DEabtSepPqzm8nS1EXDgRGM0Bcl+xkxC6r1ktUxcJ2i0jOfMmYMTJ05oLHPmzNEpn5ycjM8//xybNm1C586d1bb16dMHdnZ2sLGxwcCBA/Hmm2/i0KFD+sPVt7HBtS0iIgJ5eXno3r1749+3b9+mcVXoYWtrh159hqCirBRXLp1W23YiZS8AYNAgzRk6pPqwMqnl6qz9175zBzsEtLbBnYfUh4SR7GfMJKTeS1aLHD9joVAIf39/jUUoFGqVTkpKwooVK7B582ZERUU1GQqXy22yN8Gsrm3nTx1BybNCAKqXczKZDAd2/QwA8PRupTa5443Z7yLr+mVs/PbfGDF2Cjx9WuP61YvIvHoBEydO/H9rz+bhZ8yk1oI3POEm5OHWHQmKRTIIBBy097dF3+5OqK1VIO5AqUXiYlpv+AAP+Hiq3h24OgvA53MwY5IvAKCopA7Hz1nmPFmtZuRnbKLRFPHx8fjxxx+xbds2dOnSRWN7VlYWFAoFgoODweVykZ6ejl9++QWLFi3Sq2vWoW2n//gdubf+VFu3J0FlLh8S1l0tGXt6tcIXa7YhaedmnD1xGDU1VfBp5Y/pc9/Dv/7xrs5jkOrDypTW+YwqDO7thIE9nSF04kKpVHlT/HFRjEMny1FSpt2bwhznyKTe6CFe6Baq3ip5a6o/ACAzW0wrGTMZF6vVfPyMTfUFj6+++gp8Ph+zZ89WW5+cnAxfX18UFRVhzZo1KCwsBI/Hg6+vLz744APExMToD9ecrm1X8qi9WGqKhunQukxh6KDPeIUErZeNggxFl1GQIegzCjJES5tRkCHoMwqiC+n1gtR7yfT1NwbJyR2Uy9oPnWX08YyFtdBkYWGxSiSnEiiXtR8yw4SRUMOs3RRbtmxBVlYWsrKykJ+fDz8/P5w8eVJr2X/+85/Yv3+/1m3r169/4Tt4LFShc/0vXLiAY8eOISsrC7dv34ZUKkV8fDytGUUsLBaFEM8JqpjXtW3dOjg6uaBNuxA4lItRK1XgbFa11rKF5Srjm3nvf6mxLTw8HID2R7X3Fzy360vLEKns+sb54npWhVa7Pn2Pfdai1aC3bt062Dm6olVAKOwcKlEpAX45pVkWAA5v/x3ZVw7DyzcIrj7t8exxDlKuAdk1wNwhuh+TDTnPsfNvaei8+ZoPxg5xR3pmJfL+qoFMrkR4iBMG9nLBX48lWLrqL0jr1cWSY8N0dnmcSOzdaMkZ1M4RNRK5Xn/kE4mqyUak3UtT1AtStYxFSchXn6li1mS8auMheLVSvYT5/P0pqKutaXKfqEFjNdb5+mqaBAHkWv+RovXOl8fh6hUAAIhdOQ5SPdd/0IQPMWrGSvAFNkhP3YZnj5seWcPkeZ6/WoHdR4pRI3k+OyvlTBkKiuowbZw3RvR3w+FTIkpaAHOWnKTcy5aoRZtm1jI2a7QNiZgOSqUSkpoqKBRNT5kk1fqPFK2GREwFZzcf8AU2lMsbG9vL3H1Yq5aIGzh7ReXV/IqfHa3YmLLkJOVetkQt2pjIm8JUEOna9iLvzRyIWkk1+HwBgkK7Y+L0hRjYWfsga1Kt/0jVYhpzxObpJgAAlItlRmsZAqn3siVo0UXJmsszg4urB4aPn4EZb3+ChR9/i9GvvYWH97LxzfJ5uHjxotZ9mrLrc3OxAZ9P7VewJWgxjalj43KAmHHekMmUOJ3OzDBJupB6L1uCFm2syVy+oKAAvr6+5opFjddmvaf2d0TkEEQOGI2VS6fhiy++QGpqqsY+VO36qmRNT4xoCVpMY+rY3p7WGp06OGD73kI8KZIaHKcxkHovW4IWXZSEJFmq6I32xel7+pyLzIWPbxv07PsqHj58iPv372tsJ9X6j1QtpjFlbDMneGP8MA+knBEhKaXE4BiNhdR72RK0aNPM+oz1JuMX54M8fvzY5MFQwcNb1VIvKyvT2FYiksJFKIBAy2OPIdZ/1q7FNKaKbXq0N2LGeyP1fBk27ND9ZXBzQOq9bAladFFyuJQXEjDYQtNSPHv6CADg6empsY1U6z9StZjGFLFNj/bGjGhvHL9Qhv/FPWl6BxND6r1sCVq04fKoLwRgsIWmyjXNNNTVSlAv1RyK9OivXGRcPI727dujTZs2GttJtf4jVYtpmI4tZpwXZkR748TFMny//YlpTWUoQuq9bAladFFyOJQXEjCrheal04dRWqyy0KwUl0Euq8fhpFgAgIdXK/QZPA4AUPT0Ef735RJ0ixwM79ZtYGtrh/wHd3D+5EFwuTysXLlSqz6p1n+kaN1KO4AKkeoxv6ZSBLm8HheObAQAuLj7IixqYmPZZ/m5uHNDNVX6yT2V015W+kHk38uAJAdQyAPB5TmY7DzHDnHHzIk+eFYqRWZONQZHuqhtLxPLkJmtffamNpiy5CTlXrZELdoQ0v1AFbOOMz5/4iBuZ2WorTv4myoZdOzcozEZu7h6oFPX3si7dRXpZ1NQL62Di5snevUbgdGT56JnzzCdxyDV+o8EresX9uLxHfXpwucOrQcABAT1VkvGhY+yG7c1cOOiytj/HAD3wFXAS8mYyfPs2NYeAODtYYOl8zQnC93Iq0ZmtuZLXF0waclJwr1sqVp0UIKMFi9VzOrapsuHgi4DO6umQ5NmSUiqVoOeLh8KuujzpqCLLm8KQ9DnTUEXfd4UdCG9XpCqZSzl17SbYGnDNWKo0cczFuJn4LGwsLAYRDPrpmD9jFlYWKwS0Y1zlMu6hw8wYSTUMGvLeOGnm/H0YTaePsxCWXE+XD188eFazUeJ+vo6XL94ELczT6PocS6qxKVwdvWCX7twDI5ehCWvdwCg/VGNwwGmRPthwihftPK2Q3mFFKfOFyM24YHWweX6HvusRatBj0mtf26t1Vg/vDsfw3vorlJyhRKfbVMfJbN6gZ3OR1tDrhmTWgB595L0esGkltEQMkqCKmZNxif2fgd7Rxe0fqUzamsqdZYrL3mC37f/G22CeiBi4OtwdvVGWfFjXD2ViJyMP9DLP1bnF1nfm//cOzVx/2OVd+p4PwS1c9LpnaoLVoue1q0HcpSINf9ztXbnYlBXPnIe0ptpRep5slqW06IDKZM5qGLWZPz+13/A3Vtl4/jj8vGQ1mp/oefo7IZ3VuxH6zad1NaHR43H5i8m4ZtvvsG+ffs09iPVh7UlaAFAoUiJQpHm/6zA/qr/FFfyqPsPkHqerFbz8TNubqMpzPrT0ZCIm8LByU0jEQOAt18HePsF4c6dO1r3I9WHtSVo6ULAB7q256G8Sonb+dRbxqSeJ6tFXh3ThammQ69ZswZjx45F9+7d0b9/f3z66aca9gzZ2dmYNm0aunbtisGDByM+Pr5J3WbVjlcoFKiqKNY6FRog14e1JWjpIjyQBzsbDjJuy2k9jpJ6nqwWeXVMJyYyCuLxeFizZg3S09Nx8OBBFBYW4pNPPmncXlVVhfnz56N///64fPkyvv/+e2zYsAFHjx7Vq6s3GR8/fhwJCc+/sDplyhQMGzYMw4YNa1LYFFw9nYjK8mJMnDhR63ZSfVhbgpYueobwoFAqcfU2PYtEUs+T1SKvjulCweFRXsRiMfLz8zUWsVisofvRRx8hNDQUAoEAHh4emDVrFi5ffj6+PTU1FVwuFwsXLoStrS26deuGKVOm4Ndff9Ubr95kHBsbi6FDnw+Glkql2LNnD3bs2IHffvuN7rUxikd3/sSx31ajVUAI3nnnHa1lqHqnUoHVoqelDU8XDgJbcXGvQIGySnpvaUg9T1aLrDqmDzrdFHFxcY0NzReXuLi4Jo9z6dIlhISENP6dm5uL0NBQcLnPzyssLAy5ubl6dfS+wKuvr0fr1q0b/+7Rowfc3Nzg5uYGiUTSZJBMUfDgFhK+fwfObt6Y8eEW2Nraai1XWyeHm53277YZ4sPKahnnNdsrWOWGdSWXvnE4qefJapFVx/RB5wXenDlzMGnSJI31QqFQS+nnHDlyBElJSdi5c2fjuqqqKjg7q3e/CIVCVFXpd6jT+5P0chP93//+d+O/RSLqX+Y1hoIHWYhfOw929k548x9xELr56CxLqg9rS9B6GS4H6B7EQ3WtElkP6P9nI/U8WS1y6lhT0GkZC4VC+Pv7ayz6knFycjI+//xzbNq0CZ07d25c7+TkpJF4xWIxnJyc9MarNxmHh4dj9+7dGusTExMRHh6uV5gJVIn4LdjYOeLNj+Ph6umntzypPqwtQetlOr3ChbMDB9fuyCE3oOFD6nmyWuTUsaYwpYVmUlISVqxYgc2bN2vMeQgJCUF2drbaF+2zsrLUujK0oTcZf/rpp9i3bx9mzZqF1atXY/Xq1Zg1axb279+PTz/9lPYJ0OHpw2xVIrZ1wJv/iIObl6Zz18uQ6sPaErRepmdDFwWNscWmio3Vsg4tutB5gUeH+Ph4rF27Ftu2bUOPHj00to8YMQJyuRybNm2CVCrFjRs3kJSUhJiYGL26evuMPTw8kJiYiEuXLuHu3bsAgEGDBqFPnz60gm/g+sWDKC9R+elWV4ogl9XjzKFNAABXT1907TsBgGoGXvzat1BbI0bk8Fl4fPcaHt+91qhjXwy8+uqrGvqk+rC2BK0XcXYAOvpz8eiZAkVlhj2CknqerBYZdYwKppr08dVXX4HP52P27Nlq65OTk+Hr6wsnJyfExsZixYoV2LJlC9zc3LBo0SKMHj1ary6lGXh9+vQxOAG/yJ9n9+BB3hW1dSf3qzxz2wb3akzGZSX5qKlSfYr99MENGjr7AJw4cULrMUj1YW0JWg306MgDj8vBlVyZYQImiI3Vsg4tOphqOnReXl6TZUJDQ7Fr1y5aumZ1bUu8yMyhpvVV/eKR5g9LqlaDHpNa2oyCDEGfURBdTOGnS9q9JL1ekORn/ODubcpl23boaPTxjIW10GRhYbFK7t+7S7lsYPsOJoyEGmYzCrp//z4OHTqECxcu4NGjR6irq0ObNm0watQozJkzBw4Ozz/hExwcrFfrgw8+wLvvvmvqkFlYWJoxzc0oyGwt47Vr1yIh7hcMCvJHuL8X+FwurjwsRGr2Q3T0dkP83FGwE6h+G5Jv/qVVY/PZG3hcVomDBw8iJCRE6yPR+wue2/WlZYhUdn3jfHE9q0KrXZ++xz5r0WrQM7WWoedZcioN5emZqH1SBIVMBmFYR/jPmQyZuBrnek6AtFg1pt1verTWYwb9azEcO7yi83oZGhcArZ9xOpHYG+JKGe7cr0ZQO0fUSOSYseS61uM2lNcVW0uoF4ZqGcvde9S/kdihfaDRxzMWs7WMR44cidkuYji/MBtnSo+OaON2DbEXbuFA5l1M66Uahze2SzuN/YvE1XhSXoXQ1h46x+uRav3XErSM0UsfMUdjXem5q+iRuB7+syfjr29VXxB/8ushjXJ2fj7o+vNqlF+9CdeeXcxynjPfu46nz1Qm+bFrwmBvR29olCniagladFE2Lx8080XbpUsXtUTcwMjObQEAd4vL9e5/8Po9KJRKTO6mu2+HVOu/lqDFtJ7kkUpD4KZ/Oqr/nNfA4fHw+Ocks8QFoDERGwup95JULboowaG8kIDFfzqKxDUAAHdHe51llEolDl6/B3sBH6PC2uosR6r1X0vQMlaPa2sDgYcb7Px84Dm8H7psXAkAeJai/+18wJzJkFVVoyDxsEniMiWk3ktStejCJmMayBUK/HT+JvhcDsboSbKXHxTiSXkVXu30CpxstZuOAORa/7UELWP1AuZNwYjCNAx7cBaRKT9D4CLEtdnLUHYhQ+fxPIZEwaFdAJ7uPQpZpfavxhgblykh9V6SqkWX5paMzfrZpZdZk3oVN/KLsWRIN7T1cNFZbv811RCVSXq6KADqdn1Vsqan6LJa9LSM1Ss6eBxVuX+B7+QAYbdQ+IwfChtPN73HC3hrCgDg8S97TRaXKSH1XpKqRRdSkixV9CbjL7/8Ehw9JhrLly83+MA/ns5E4tU8vBYRhHn9tL94AYAKSR1O5j1CoIcQEW309y+Rav3XErSM1at9UoTaJyqfgqJDJ1C4PxX9Lu0Bz94O9775SaO8wM0FrSa+iqqce3pbz8bGZUpIvZekatFFobR4Lywt9EYbFhaGzp07o3Pnzjh58mTjvxsWQ9l05jq2nr+JCV3bY/mYSL1lj9y6D6lcgYlNtIoBcq3/WoIW03qVN/MgzszGK+9M17rdb/p48Oxs8fiXPWaNi0lIvZekatGluXVT6E3GkyZNalyEQqHa39qMmKmw6cx1bDl3A+PD2+HzcX30trwBYH/mXfC5XIwP1xzu9jKkWv+1BC1T6PHs7SBw19595T/3dSikUuTvPGD2uJiC1HtJqhZdrCoZv0hTSZMKW87ewJZzNzCuSzusGN8X3CY0swpKcbuoDIM6+usdbdEAqdZ/LUGLaT2PQZFw7hyE8nTNyRQuPcLg0rUTig6fapwQYq64mITUe0mqFl2USg7lhQTM9gIvISEBm85eR2uhIyIDWyHllvrsGHdHO/Rp56u27kAmtRd3DZBq/dcStIzR63t+F0pPpUHyqABcO1u4dO8M3zfGQFZZjZx/rNYoHzD3dQDA45+b7qIwxXkOH+ABH0/Vp79cnQXg8zmYMUlVd4tK6nD8XKnZ42oJWnRRENLipYreZBwREdHYIq6trUX37t0BqMb9cjgc/Pnnn5QPdPPmTQDAU3E1/nXoosb2Hm181JJxbb0MKVn30UrogL7tfTXK64JU67+WoGWonrSkDH4zJsDGyx1QKiF5WIBHW3fh3rfbUPv4qVpZrp0tfKeOheRRAYpTz1nkPEcP8UK3UPXH7remqj5+kJktppyMmY6rJWjRobm9wDOra5tkx38Y0bGfpRrFQZolIalaDXqkaiUL9BtDUWVsfR7jFo7avCnoos+bgi4tqV4YS8Zt6t/p7NHR3ejjGYtFxxmzsLCwmApS+oKpwvoZs7CwWCVX8vT73bxIr2BXE0ZCDbO2jN/+eAsKH2Wh8FEWKkryIXT3w8JVJ7WWvZ99AXnXjqHwURaKn9yGXCZFzIfxeCU4EnOHqMpoeyTicIAp0X6YMMoXrbztUF4hxanzxYhNeKB1cLm+xz5r0WrQ0/bIHTOhNYICHREU6ABfHzsUFtfptYMEVI/duh5HDTnPsfNvaaz39bHB0ChXRIQ6obW3DQQCDgqfSXE+Q4wDf5SgTqrZhkiODcPr72var/p6CzBlpCsC/W3h5sIHnweUlMnwZ3YNDp6sQLlYc/bXnvWqoZSk3cvmWC8MjctYmlvL2KzJ+MyBdbBzdEWrgFDU1VTqLZt1+XdkXzkML98geLRuj2ePcygd4735z71TE/c/VnmnjvdDUDsnnd6pLVlrfkxAozevk6Px1YGp2Eb0d8PYIe5Iz6zE6fRyyORKhIc4YfYkH/TvKcTSVX9BWk9NzMOVB1chH5dvVqO0XA65XIk2vjYY3leIft2dsOybfIirqM8CI/VeklovmIyLDqaaU5mcnIyEhATk5uaiurpa45t4wcHBsLW1BY/33Fo1MTGxyY9mmDUZv/Plcbh6BQAAYleOg7S2RmfZQRM+xKgZK8EX2CA9dRulZEyqDyupWgBz3rxMx3b+agV2HylGjeT5f6mUM2UoKKrDtHHeGNHfDYdPUXtBc/N2LW7efqqxPudeLZbO9cGQ3s44eLKCkhap95LUemFJP2NTjaYQCoWYPn06amtr8dlnn2kts3XrVkRG6p9d/DJmHfvRkIip4OzmA75At0ObNkj1YSVVC2DOm5fp2O4+rFVLxA2cvaJKmq/42RkXLIBikerr1Y4O1P8bkHovSa0XFvUzNtGkjwEDBmDcuHEICKCez6hgVaMpSPVhJVWLacwRm6ebAABQLpbR3lfA58DOlgMBn4OAVjaYGa0aznQtW0JZg9R7SWq9sLSfMVXEYjHEYrHGeqFQCKFQ/wcOtLF06VLU19fD19cXMTExeOONN5rcx6qScVPeqeGhLuDzOZSMSVqCFtOYOjYuB4gZ5w2ZTInT6dTflDcwrI8z5r/u2fh3UWk91sc/Q85ftZQ1SL2XpNYLS8aloCEZFxeHDRs2aKxfvHgxlixZQuu427dvR0REBLhcLtLS0rBs2TLIZDJMn67d9KoBq0rGpPqwkqrFNKaO7e1prdGpgwO27y3EkyIp7f0v36jGk6J62NlyEOhvi15hDnB2otdTR+q9JLVeNBc/4zlz5mg1PzOkVdynT5/Gfw8cOBBvvvkmDh06ZFwyfnE69IsYMh3aHJDqw0qqFtOYMraZE7wxfpgHUs6IkJRSYpCGqEIOUYWqS+LKzRqkX6/G6qV+sBVwsf84tZY2qfeS1Hphybjo9AUb2h1BBS6XCyrTOfQm42vXrjEWkDkoEUnRNsARAj5H47HIEB9Wa9diGlPFNj3aGzHjvZF6vgwbdhQwFS4eFkhxP78OI/sLKSdjUu8lqfXCknHJTTTOWC6XQyaTob6+HgBQV6d62SkQCJCTkwOFQoHg4GBwuVykp6fjl19+waJFi5rUbV5OGk1Aqg8rqVpMY4rYpkd7Y0a0N45fKMP/4p40vQNNbAQcONEYTUHqvSS1XljUz9hEoykOHjyI8PBwzJs3DwAQHh6O8PBwXLlyBUVFRfjHP/6ByMhIREVF4euvv8YHH3yA2bNnN6lrVcmYVB9WUrWYhunYYsZ5YUa0N05cLMP3258YPDnA1Vn7GNnOHewQ0NoGdx5SH8ZF6r0ktV5Y1s+Y+kKHyZMnIy8vT2OJjIzE0KFDkZKSgmvXruHq1as4dOgQYmJiKOma9QXerbQDqBCpHjNrKkWQy+tx4chGAICLuy/CoiY2ln2Wn4s7N1RTpZ/cU/VNZ6UfRP69DEhygFmzZmnok+rDSqoWwJw3L9OxjR3ijpkTffCsVIrMnGoMjlT/4keZWIbMbN1fhH6RBW94wk3Iw607EhSLZBAIOGjvb4u+3Z1QW6tA3AHLnCOpWgCZns10IeULHlQxazK+fmEvHt9Rnwd/7tB6AEBAUG+1ZFz4KLtxWwM3Lqq+AnwOQHR0tNZjkOrDSqoWk968TMbWsa3qyy7eHjZYOs9fY/uNvGpkZt/XWK+N8xlVGNzbCQN7OkPoxIVSqfKm+OOiGIdOlqOkjN6bfFLvJan1wnJ+xqbTNgVmTcYzlu6gXDa872SE952sdVuDURBwT2ObQgEkHshH4oF8AyJseVpLV+YarfEiTMX23S9P8N0vzPQRX8qsxqVMaq1oKpB6L0mtF0zGRe+4zatlzFposrCwWCWH/6Q+S3Ncd8tPubB8BCwshLJlyxZkZWUhKysL+fn58PPzw8mTmpav+fn5GDZsmF6tNWvW6OxaYzENza2ZadZkrM1rtktHO0x+1Q13Htaq2RsOiXSGpFah1d5Qn9fs+wue2/WlZYhUdn3jfHE9q0KrXZ8+f1hr0WrQM7WWoedJqta6devg5CxEu/YdUV5eAWm9HDfvar79r62V4b2ly7UeN3bz96iX1qF///5afYNPJPZutKoMaueIGolcr2+wvk84kVovDNUyFtbPmCasvaF1aJEcm6Fax48fR3mdavbYhwvnoFai3VDIzs4eA4eO0Fifl3MLNdVVGDlyJNzdtX9jjUSrSlK16NLcXuARO86YtTdsXlokx2aolrEWiSeOHQYATJkyRWcZEq0qSdWii6nGGZsKi7eMG2DtDZu3FsmxWcLGUSKpwcXzp+Dl3Qr9+vVjXP9lSL1elrTQNNV0aFOhMxnrMgkCABsbG7Rp0wYffvihmkORMbD2hs1bi+TYLGHjePHsSdRKJIiePA1crukfQEm9Xpa00CSlxUsVnclYn0mQXC7HnTt3sGzZMhw+fJiRQFh7w+atRXJslrBxPJGaDC6XiyHDxzCmqQ9Sr5dFLTStJRnrg8fjISQkBDNnzmQsENbesHlrkRybuW0cHz96gNu5WejWvTe8vH0Y09UHqdfLkhaaimbWTWHU89O0adOYikODF+0NqVIiksJFKICAr3kTDLEkZLXo2RuSGhvT59kUJ1NVT4vDRo5jTLMpSL1e5r72L9LcXuARO5oCYO0Nm5MWybGZ08axvr4eZ06mQujiil5R/RnTbQpSr5clLTTlCuoLCVg8GbP2htahRXJs5rRxvJp+AeKKcgwaOgJ8vvkGK5F6vSxroWkaP2NTYfGhbay9oXVokRyboVoHDhzAtVu3AQDiinLI6uuxJzEOAODl3QqDho7U2OdEajIAYNgIal0UJFpVkqpFF1K6H6hi8WTM2htajxbJsRmitXfvXly+rD6FOXHHNgBAaJduGsm4pLgIN65dQXCnMPi3aUspLlKtKknVokNzm4Fn8WTM2htajxbTepbW2rFjh1YvCl14evlg9++nacVFqlUlqVp0YFvGLCwsLATQ3JIx62fMwsJilWw9Tr3sguHUyyYnJyMhIQG5ubmorq5GXl6e2vbs7GysXLkSOTk5cHNzw1tvvUXpg6Rmaxnfv38f+9Z9gUt5D5FfUo46mQwBHq54tVtHzBjYAw62gsaym45exOZjl7TqfBQ9EH9bsxWAdhtBDgeYEu2HCaN80crbDuUVUpw6X4zYhAdaB5frs6q0Fq0GPVNrGXqeY+ff0ljv62ODoVGuiAh1QmtvGwgEHBQ+k+J8hhgH/ihBnVSzDZEcG8ZoXAC02l7GTGiNoEBHBAU6wNfHDoXFdZRsL+uePALfzQMcHg/1JcWovnYZot/3QF4uaizrNu41OHWPhI2vP7hOzlBUVUJakI+ylAOounIRwbuOAWhe9cJQLWNRmGjImlAoxPTp01FbW4vPPvtM+aWVkwAAGWBJREFUbVtVVRXmz5+P6dOnIy4uDjk5OXj77bfh7e2NUaNG6dU1WzLeu3cvEs5kYHDn9hjTPQR8Hg9X7j7ChiMXkJp5Gzvej4GdjUBtn79PHAxXR3u1daEB+mc0vTf/uXdq4v7HKu/U8X4Iauek0zuV1WJGi0m9Ef3dMHaIO9IzK3E6vRwyuRLhIU6YPckH/XsKsXTVX5DWUw+OyfOcHxPQ6EHs5Ej9v1DV5YuQiYqhlMth2yYQrsPHQNh3MB58/C7kYpVNrF37YNQXF6E68wpk4grwnIRwjhoAv2Wfo2RXnNnOkVQtOphKd8CAAQCA9PR0jW2pqangcrlYuHAhuFwuunXrhilTpuDXX38lJxmPHDkScwJt4Wxv27jujX5d8YrXeWz9Ix37028hZkCE2j5DunSAn7vLy1I6IdWHtSVoMa13/moFdh8pRo3kefMm5UwZCorqMG2cN0b0d8PhUyI9CqaJCzDcg7gk8Re1v2tybsLvw+VwGTwCokNJqpjWr9LYr+zIPryy+ke4R+u24iS1XljSz5hOMhaLxRCLxRrrhUIhhELqs4Bzc3MRGhqqZg4VFhaGpKSkJvc126SPLl26qCXiBkZ2CwYA3H1aonW/qto6yChOkSHVh7UlaDGtd/dhrVoibuDsFVUL8hU/O4vEBTDnQSwrViUhrqOT/oIKBWSiEnBsdZ8zqfXCkn7GCiX1JS4uDsOGDdNY4uL0P428TFVVFZyd1W1BhUIhqqqanmlo8dEURRWqID2cHTS2TfkmHtV1UvC4HIS1aYW3R/RB/06BOrVI9WFtCVqm0NOGp5uqK6tcTP1jk5b01H0RjkAArp09OAIb2Pi3gdf0eQCA6mtXNMpyHZ3B4XLBEwrhHDUQjl17oibrOhy7RGiUBcitF5a89nTGJsyZMweTJk3SWE+nVQwATk5OKC1VHxsuFovh5NTEDy6aSMZPnz5F69attW47deoUhgwZQiNMTeQKBX5KvQQ+l4sxPTo1rne2t8VrfcLRra0vhA62ePCsDDvP/onFW/dhxbSRmKrDlZBUH9aWoGUKvZfhcoCYcd6QyZQ4nU7Nyc8ccVHFZeho+Ly1qPFv6bNCFPywGpJczReYgd9vA1+o6qJTymSounweRbEb0CF2t1ZtUuuFJa+9nMZ8MbrdEboICQlBSkoKFApFY1dFVlYWQkJCmtxXbzKeO3cuYmNj4e/vr7Z+z5492Lx5s9HJ+Jv9p3D9wVO8N7Y/2no//0bYzEE9NMpOjAzDa1/HYc2B0xj392o4OjpqlCHVh7UlaJlC72XentYanTo4YPveQjwpklLez5Keui9SdeUipAWPwbW1g21gBzj1iALPWfs7kYJ1K8ER2IDv7gnnqAHgCGzBtbfXWhYgt15Yo5+xXC6HTCZDfX09AKCuTtVtJRAIMGLECKxduxabNm3CggULkJubi6SkJHzxxRdN6urtM/7kk08wb948PHjwoHHdli1bEBcXh507dxp+NgA2HLmAxPOZeK1POOYNj2yyvKujPab0DUelpE6n8X1tnRwCvvZTMsSHldWi72fMpN6LzJzgjfHDPJByRoSkFO3vFywRFx1kohLU3LyGqquXUJq0A4Ub18Jrxjy4T5yqUVaScws1N/6E+HQqnqz+FxS1NWizcp1ObVLrhSWvPZ0+YzocPHgQ4eHhmDdP1c0UHh6O8PBwXLlyBU5OToiNjcXZs2fRs2dPLFmyBIsWLcLo0aOb1NXbMh40aBBsbGywYMEC/Pjjj0hKSsLNmzexc+dOuLhQH+XwMpuOXsTWP9IwoXdn/GsK9dHWvv8/sqKsrEzr9hKRFG0DHCHgczQeiwzxYWW16PsZM6nXwPRob8SM90bq+TJs2FFAe39TxWUsdY/uo+7BPbi+Oh6iA7v0lq04cxzCfrqfREmtF5a89qZqGU+ePBmTJ0/WuT00NBS7dum/n9pocjRFnz598N///hezZs3C48ePERcXZ3Qi3nzsEqJ7dcYXU0fq/M6eNh6VqJKwp6en1u2k+rC2BC1T6AGqRDwj2hvHL5Thf3FPmt7BTHExBcfGFjynpl9icW20fy2jAVLrhSWvvVKhpLyQgN5kHBERge7du2PBggWorq5GWloa+vTp07ieLpuPXcLmY5cwrmcoVkwbCS5XMxHL5ApUSjSHDhWWibH7wnW4OtohIkL7G2VSfVhbgpYp9GLGeWFGtDdOXCzD99ufGNzSsaSnrj7sO3eFbcArkNzJAQBwbG21D1/jcOE6MlqvFqn1wpLXvrmZy+vtptD3UVK6JCQkYNPRi2jt5oyojm1w5M8cte0ezg7oE9wWNVIpxnwZiyFdOqCdtzuEDnZ48EyE/Wm3UCOVYvWssbCz0z7eklQf1pagxbTe2CHumDnRB89KpcjMqcbgSPWnsTKxDJnZ1Nz+mD5PQz2IPafNRX1JETgCG9i1C4Kw7yAoJBIU7/gJAGDTyg8BX6xFZdo51D/Nh7yqEnx3Dzj3HQJbvwBUnE6Fy+ARJj9HUrXooiCkxUsVs40zvnnzJgDgaVkllv96VGN7z/b+6BPcFnYCPoaHB+Hmo0KcunkXkrp6uDraI7JjG7w5tBe6vKJ9qF0DpPqwtgQtJvU6tlWNHPD2sMHSef4a22/kVSMz+77Z4wIM9yAWDhgGntAFgBL1xc9QfvwIRIeSICstBqB6wSc+ewL2ncLg3LsfuHb2kNdUo+7BPZTuS0Dl+VM6kzHT50iqFh2amwWaWV3bao/8xIiO3Zi3AWg3S6GLPkMea9Fq0CNVS5tRkCHoMwqiiz6jILo0GAXlTdX8Mghd9BkFGQLJ9cJYvkqkPlzus2nUprSbEtZCk4WFxSr58jfqszT/FWPxyciWnw7NwsJCjy1btiArKwtZWVnIz8+Hn58fTp48qbP89evX8d133+H69evgcDiIiIjAsmXL0KlTJ537WANKQl7MUcWsyfj99ZVa17s5c7DkNQfYCoC07Ho8K1PA3pYDX08ech7KcO22+i/c+vdVQ4G0PRK9v+C5XV9ahkhl1zfOF9ezKrTa9enrDiBBS9tj8onE3o0WjkHtHFEjkev1023Yh4m4GmLT9ThqyHmSqgWQWS/WrVsHOwdXePmHwtahEuIa4McUraeNpw8ysW/DLDi6+KDHiPcBANfP7cSUqdMx5f1EfD4v2OT1wlAtY5HLm9dDPxEt41kj7cDlAl8n1EBc8+IFrKelQ6r1HykWjqaOi9RrZm1ax48fx6FbAQCAnavHob6uRucxzu77D7g8AV5fkgAnV5UXeFC30djx39E4d/BrYN7PRJ4jEzS3HlizWWjqor0vD+39+DiRIYW4RgkuFxAY+BNBqvUfqRaOJFtoslq6tQICAijplxc/RNGjmwjqNqoxEQOAk6sPgrqNwuPbF1FcXMxYXNpoLhaaJGDxlnFooKpVV1apxILx9ujUlgcel4NnZQocS6/D1TzLWCWSqsUkJFtoslrG14uiR6rhpK3aak6SatW2G7LT9yIrKwuA+uSr5nSO+iBlZh1VLN4y9nZVhTBtmC0c7DhISK3Fr39IIJcrMWuUPSJDqf9eNGXX5+ZiAz6f2vRrUrWYhOm4SL1mLUFLG9Vi1eO/k4tm69PJRdVSLirSnAHXnM5RH0ol9YUEDE7G27dvZyQAWxvVjaiTAhv21iAjT4b0bBnW76lBTa0SY/vaguqtomrX15y1mITpuEi9Zi1BSxsyqQQAwONr+lo0rJNIJCaNy5J1X6FQUl5IwOLJuP7/eyEybterzRGX1AG3/pLBxZELbzdqYZJq/UeKhePLkGyhyWoZXy/4NqpZjHKZpvdzwzp7LR7Jzekc9aGQKykvJGBwMmbqTWV5lepGVFZr6olrVNvsKX7urEQkhYtQAIGWxx5DrP9I1GISpuMi9Zq1BC1tOApV3RNVFZqjFaoqVN0TPj6aX1tvTueoD4VSSXkhAYOTMR3rS308KlJNWXR11tRzdVKFV1VD7WKRav1HqoUjyRaarJbx9cKnTRcAQOEDTcOvwgeZAIeDzp07mzQui1poKpWUFxKgZKH58hIREYFnz5gZG3jjngy1dUr0DBbARvB8vdCBgy7t+CgqU6CkgtrFItX6j1QLR5ItNFkt4+uFq9cr8A4Iw53Mo40tYUDVKr6TeRQBQVHw8vIyaVyWrPvNrc/YbBaaupDUAQfO12HaMDt89IYD0rLrwedx0K+LADwesPd0LWUtUq3/SLFwNHVcpF4za9M6cOAALp9RffFEUiWCQl6Py6kbAQDObr7o1GtiY9lBkz/D3g2zsed/M9B1wEwAqhl4UCrRf8I/iT1HJiCkwUsZi48zBoBLt+pRLVFiWA8bjOljC6USePBUjvijUtx/Su9DhaRa/5Fg4WjquJjWY7W077N3715cvqw+TT7tyHoAgF/73mrJuHVgd7y2eAcuHfleVYYDtG7bHWPeXA8vP91fLLb0OTKBnBTXeIoQkYyB/2vvTIOiONM4/u+BKDAwCqIcw8phILIJEEAJMVtYVC1FIl6YjXHdRdSQKi1go7vi+kErCRWqJCbqFhrE4FqK1E4lIkJqzUaNV4JJeYJCEoLuoOKJAs6ggDI9+2F21HGu7rl4ZZ5fVX/p6fn3M0c90/3O27/WDVecuyj8Ag9z8Dyg2NsBxd6OYZv1t+Jf7M7Q48i6HJ1HWaapqqoy66IwRUhkIubk73B6Xa7IEsOzdtEHM82YIAjCkTirGZeVleGzzz4zuONQeno61q83f/duIZDPmCCIYcnSdT2Cty0vGi1427KyMpw4cQJVVVW2lGUWlx0ZK5VK7PmoCD8or6OjW42BQQ3C/P2QERuOP6dMhPcTUyle/sjyi1y2bBmWLl1qUtfHccBbM+WY9Xoogsd5oefuAxz+vhOV1e0mJ5db0hsOlyx9nrOzbH2drGYB7H2WzvheLPxrBTo7WnCrowWqOx3w85dj0fvm/cg32pvww74NuHGpCQCHkMhEvDZ9BT54N9ahddmLmCNjlUoFlUpltF4mk0Emk5l4huNxWTOuqalB9YmfMTU6DNNeioSnRIKTl25g85FGHPjpEnYueh1e/9e1lcx6zWTGlmPncKVbjfT0dLP7+UveY3eqovaKzp06Q47oKF+z7lTKckwWy7VRlvmsH/792I880GfaOa7nST9y6huP/ci7y+bjj2kKp79GMYg56d+xYwc2bdpktL6goACFhYVG65ubm5Gamgpvb28kJSVh2bJlgm165nBZM87MzMSCUSr4eT2+Tv6t5BiM9z+LyoZm7G28gHmTdf/uZsVFGT3/puoervb04rchYzBxoul/gVn1sLpDFsu1UZblrNzVBzEq0H4/cmlpKYBcp71GsYiZTZGbm4vs7Gyj9aaOijMzMzFnzhyEhobi1q1b+PTTT7Fo0SLU1dVBKpXaXK/L7DRxcXEGjVhP5osRAIALnZbHd+qaLoLXajHn5efNbsOqh9UdsliujbIsZ+kbsTWs+ZGPHz8OfvCuw+qyFy2vFbzIZDKEhYUZLaaacUxMDORyOTiOQ1BQEEpKStDZ2Wn3dRlDrtC8qdL9CgdIjYUlerRaLeqaLsL7OU+8/lKE2e1Y9bC6QxbLtVGWY7zB1vzIWq0WD/svubwuc4hpxvbAcRw4jrP7suohbcYansfW78/DU8JhmoUme6L9Bq729CIjNhy+I42PrvWw6mF1hyyWa6Msx3iDhfiR+UHDM9yh9Bk7SxS0b98+dHV1AQDu3LmDNWvWICAgAImJxj9SYrA4ZrxkyRKLT96yZYtdO1+3/xTOdXSiMP1lRIwZZXa72rMXAADZFoYoAOHu1N5B61f1UZa4LJZroyzxn6UphPiRtVpDXacr6jKHs+YZ19fXo7i4GH19fZDJZJg8eTK2b98OX19fu3ItNuPGxkaEhIQgKysLCQkJDrUbbT7SCMWpVryZGI13Xoszu93dvgEcar2MyDEyJI63PL7UP6CBv4lxacA2DytlifcZs1gbZTnGGyzEj8xxhjW4oi5zOOsSCnsPQs1hcZiioaEBy5cvR1tbG0pKStDQ0AB/f3+kpKQgJSXF5p2WH23C59+fx6yECVg97RWL2+5rVuKBhsdsK0fFALseVnfIYrk2ynKMN1iIH1niaXjxxFD6jDWDvOCFBSw2Yw8PD6SlpaG0tBRffPEFwsPDkZOTg127dtm8w/KjTaj47hxmxEfh/emvWvUi1zZegKdEghnxxtPdnoZVD6s7ZLFcG2U5xhtszY/McRye8wp3eV3mGFY+YwB48OAB9u/fjxUrVqC6uho5OTnIyMiwaWcVx86h4rtzmB4XhQ9nTIHESiNuuXYHv97sxtSYMIuzLfSw6mF1hyyWa6Msx3iDrfmRU1NTIfE0/O9nKH3GWp4XvLCAxTHjlStXoq2tDWlpaSgoKEBMTIzNO6qurkb5sSaEyKR4JTIYXzcrDR4PkHrh1ahQg3V7G4X9caeHVQ+rO2SxXBtlWc76+eReqLvt9yOvWrUKeUWGzXUofcasSOOFYrEZ19fXw9vbG+3t7QZSDK1WC47jcObMGcE7On9eN0fxuuoe1tQfN3o8eXyQQTPufziIr1uUCJb5YMqEUKPtzcGqh9UdsliujbLMP++nH2tw9aL9fmTdlbHGR7pD5TNmZfhBKC61tvVVfeSQHO+c1QBMy1LEYkm8Mlyy9HmUJS4LYO+zdMb3Qowb2RL5bzi2Lnv5w3v/Fbzt7n9Y/0/K2ZDPmCCIYQmvZWMsWCjkMyYIwmFUVFSgpaUFLS0t6OjogFwux6FDxjrOgYEB1NXV4fDhw2htbcXt27cxduxYJCQkID8/HxMmTLC7luyCNsHb1m6Ktnt/9uLSI+NVnxvfXDQ4gENwgPGsipAACaYmeKJZqcGugw8NHlv7rs6wb+qU6L13H+v6fjzdpdP1TQ9FU8tdk7o+S6d9wyVLn+fsLFtfJ6tZAHufJevfi/Xr18NbOgoh4S/CW6rGvX4tFMeNN+683oFNa9ZgfHQyYlLeRPLocejuvIJDhxX4zzf78c9tlUhNTTXeiQjotksiudGlxY0u4zct8ne6WXcnW4VfJsmKktAds1iujbJcl/Ve6QEEjNNZ4DavnoEH/fdMbif188eSD2sRMj7WYH186gxs+SAbH3/8Mfbs2SOoXnM8ayf9Q25tM8VznkDCBA/09Grxa4fwcR9WlITumMVybZTluix9I7aGj6+/USMGgHHy5zFOHo22NuFDDObgeV7wwgJDfmRsivhID3iN4NDQPChq6gurSkJ3yGK5NspiT8dpDp7n0Xu3E4GBgfZnaRwvH3ImTB4ZT5roAV6rxalfxb2ZrCoJ3SGL5dooiz0dpzlOHVFA3dOJ2bNnW9/YCq7yGTsK5ppx4CgOkcESXLzGo1st7k0SquujLMdnsVwbZQ3t90Iol9vO4Jt/rUXwbyZa1fcK4VlrxhaHKUzdoE8Px3HIz893eEGTX/AAAJz8RfwpBqtKQnfIYrk2ymJPx/k019qbUb1xCfz8x+FPyyswcuRIuzOftXnGFn/efHx8jBaO41BTU4PKykrHF8MBSdEeuNevRUu7+DeSVSWhO2SxXBtlsafjfJJr7S3Y+ck78PL2xcKVOyDzD7L+JAE8a0fGFpvx4sWLHy1z585Ff38/ampqMG3aNBw8eNDhxcSGS+Dnw+FsmwYibuz6CFaVhO6QxXJtlMWejlOPrhEvxggvKRb+fSdGB8qtP0kgz5q1zerAT09PDzZs2ICZM2dCo9GgtrYWRUVFGDNmjMOLmaQfohAxt/hJWFUSukMWy7VRFns6TgC4fuknXSMe6YOFK3fAf2yYQ3L18BqN4IUFLI4Zl5aW4sCBA5g7dy6++uorSKVSpxXi5wPEhElw+RaPm922nTawoiR0xyyWa6Ms12U1Ha9Dz22djvOeuguawYc4Wl8OABgdGIqEKbMAAD23r2LnJ4vRf1+FV36fgysXzuLKhcfSeu9OICMjAz4+PoLrfpphpdDcvn07RowYgfLycoP7Ptmi0LRGcowHPCQcTv4yaFcOC0pCd81iuTbKck3WmWO70d560mDdoVqdjjPihcmPmnH37Q7c79XdSfpInfFEgT0Avv32W7uasTOHH3iex8aNG7F792709fUhKSkJxcXFkMttH2ZxqSjIlJvCFiy5KcTCqt6QFJpDnwWw91my/r0w5aGwhXlT7J/HPHWOsTfdHEf3TBGVvXXrVigUClRWViIoKAhr165FY2Mj6urqIJHYNu2PuXnGBEEQjkCr5QUvYlEoFMjLy0NUVBSkUimKioqgVCpx+vRpm+slhSZBEMMSMUf8+6oToVKpjNbLZDLIZIazStRqNSZNmoQvv/wS8fHxj9ZnZWXh7bffxoIFC2yql0k3BUEQhL2IuVtIWVmZyYvcCgoKUFhYaLCut1c3te/pJu3n5/foMVugZkwQhNuTm5uL7Oxso/VPN1wA8PX1BaA7Qn4StVr96DFboGZMEITbY2o4whx+fn6Qy+Vobm5GXFwcAF0jvnz5MmJjjbWgQqE/8AiCIEQyb948bNu2DUqlEvfv38e6desQERGB5ORkmzPpyJggCEIkeXl5UKvVmD9/Pvr6+pCcnIzy8nKbp7UBNJuCIAiCCWiYgiAIggGoGRMEQTAANWOCIAgGoGZMEATBANSMCYIgGICaMUEQBANQMyYIgmAAasYEQRAMQM2YIAiCAf4H3UTgrhiDkvYAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fg-boTmCBUP8"
      },
      "source": [
        "Note: \n",
        "\n",
        "* `macro avg` corresponds to the average of f1 scores per class **without considering the proportion for each label** in the dataset.\n",
        "\n",
        "* `weighted avg` corresponds to the average of f1 scores per class **considering the proportion for each label** in the dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L5XkLK_TzoFv"
      },
      "source": [
        "# 3 Models 2 to 7 = LSTM (with and without previous word-embedding)\n",
        "\n",
        "For this second set of models, we used LSTMs, artificial RNN architectures with feedback connections allowing a contextual representation of tokens. A LSTM uses an embedding layer. For **Models 2 and 3**, this layer is initialized with random weights trained simultaneously with backpropagation as any other weights. Model 1 uses fixed-length inputs whereas Model 2 accepts variable-length descriptions of jobs and is more performant (but increases the training time).\n",
        "\n",
        "**Models 4 and 5** are respectively the same as Models 2 and 3 but we initialized the embedding layer with the pretrained word2vec embedding used for Model 1 and. For **Models 6 and 7**, we do the same, but we freeze the updating of the word-embedding weights during the training. \n",
        "\n",
        "\n",
        "Finally, we tried to improve the performance of Model 7 (best LSTM model) by **tuning** (Probst and al., 2018) learning rate which is known to be one of its most crucial **hyperparameters** (Greff and al., 2017).  \n",
        "\n",
        "\n",
        "\n",
        "**Bibliography used for LSTM** \n",
        "\n",
        "article: https://towardsdatascience.com/multiclass-text-classification-using-lstm-in-pytorch-eac56baed8df\n",
        "\n",
        "code inspiration: https://jovian.ai/aakanksha-ns/lstm-multiclass-text-classification\n",
        "\n",
        "understanding nn.embedding parameters: https://discuss.pytorch.org/t/how-should-i-understand-the-num-embeddings-and-embedding-dim-arguments-for-nn-embedding/60442/3\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        },
        "id": "kGIgPZwNf8VF",
        "outputId": "c8f2f31b-e92f-4861-8859-b010953dde61"
      },
      "source": [
        "# We keep only the relevant columns : we want to predict the major code (grand domaine)\n",
        "# using the description\n",
        "y_variable = \"major_code\"\n",
        "data_lstm = data[[\"description\", y_variable]].copy()\n",
        "data_lstm"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>description</th>\n",
              "      <th>major_code</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Réalise des travaux mécanisés agricoles, sylvi...</td>\n",
              "      <td>A</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Tracteur, moissonneuse-batteuse, ensileuse, ép...</td>\n",
              "      <td>A</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Depuis la cabine de sa machine, le conducteur ...</td>\n",
              "      <td>A</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Réalise des opérations de coupe et d''entretie...</td>\n",
              "      <td>A</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Avec ses cordes de rappel, son harnais, ses mo...</td>\n",
              "      <td>A</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1242</th>\n",
              "      <td>Surveille et coordonne les mouvements des trai...</td>\n",
              "      <td>N</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1243</th>\n",
              "      <td>Le technicien de la circulation ferroviaire SN...</td>\n",
              "      <td>N</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1244</th>\n",
              "      <td>Réalise des opérations de mise en fonction, de...</td>\n",
              "      <td>N</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1245</th>\n",
              "      <td>Sur les pistes de ski, une panne d'une télécab...</td>\n",
              "      <td>N</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1246</th>\n",
              "      <td>Réalise des opérations de formation de trains ...</td>\n",
              "      <td>N</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1247 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                            description major_code\n",
              "0     Réalise des travaux mécanisés agricoles, sylvi...          A\n",
              "1     Tracteur, moissonneuse-batteuse, ensileuse, ép...          A\n",
              "2     Depuis la cabine de sa machine, le conducteur ...          A\n",
              "3     Réalise des opérations de coupe et d''entretie...          A\n",
              "4     Avec ses cordes de rappel, son harnais, ses mo...          A\n",
              "...                                                 ...        ...\n",
              "1242  Surveille et coordonne les mouvements des trai...          N\n",
              "1243  Le technicien de la circulation ferroviaire SN...          N\n",
              "1244  Réalise des opérations de mise en fonction, de...          N\n",
              "1245  Sur les pistes de ski, une panne d'une télécab...          N\n",
              "1246  Réalise des opérations de formation de trains ...          N\n",
              "\n",
              "[1247 rows x 2 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 93
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kb51HnpghuT1",
        "outputId": "10e55b04-7440-4547-c14f-c60f64f0d2ed"
      },
      "source": [
        "# we change the grades to 0-numbering because it is the classical way to do to perform the models\n",
        "tag_to_ix = {}\n",
        "i=0\n",
        "for code in data_lstm[y_variable].unique():\n",
        "  tag_to_ix[code]=i\n",
        "  i += 1\n",
        "print(tag_to_ix)\n",
        "data_lstm['label'] = data_lstm[y_variable].replace(tag_to_ix)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'A': 0, 'B': 1, 'C': 2, 'D': 3, 'E': 4, 'F': 5, 'G': 6, 'H': 7, 'I': 8, 'J': 9, 'K': 10, 'L': 11, 'M': 12, 'N': 13}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "krxAHb8PLXid"
      },
      "source": [
        "key_list = list(tag_to_ix.keys())\n",
        "val_list = list(tag_to_ix.values())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pn5CRkdZi6cH"
      },
      "source": [
        "# we tokenize using tweetTokenizer because the descriptions are quite short\n",
        "#tokenizer = TreebankWordTokenizer()\n",
        "tokenizer = TweetTokenizer()\n",
        "\n",
        "def tokenize (text):\n",
        "  tokens = tokenizer.tokenize(text)\n",
        "  tokens = list(map(lambda x: x.lower(), tokens))\n",
        "  return tokens"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vj_6WMs7jHLq"
      },
      "source": [
        "#count number of occurences of each word\n",
        "counts = Counter()\n",
        "for index, row in data_lstm.iterrows():\n",
        "    counts.update(tokenize(row['description']))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ctS9iqdHnz5i"
      },
      "source": [
        "Then, we create a vocabulary to index mapping and encode our description text using this mapping. \n",
        "\n",
        "**Note :** We choose the **maximum length of any description** to be 130 words (function `encode_sentence`) because the average length of description was around 160: putting this value too high dramatically increases the computational time and deteriorates the quality of the models.\n",
        "\n",
        " The idea behind this result is that the first words contains the most relevant information to describe a job, which seems to be the case for our description (journalistic style for Onisep)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Mn4NO8ABHlM_",
        "outputId": "2bde4210-8af3-4d39-8672-e9182deb10d8"
      },
      "source": [
        "len_description = data_lstm.description.apply(lambda x: len(tokenize(x)))\n",
        "print(\"mean:\",np.mean(len_description))\n",
        "np.quantile(len_description,q=[0,0.25,0.5,0.75,1])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "mean: 126.00481154771451\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ 23.,  69., 127., 177., 346.])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 98
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "brMn9xfBkrKD"
      },
      "source": [
        "#creating vocabulary\n",
        "vocab2index = {\"\":0, \"UNK\":1}\n",
        "words = [\"\", \"UNK\"]\n",
        "for word in counts:\n",
        "    vocab2index[word] = len(words)\n",
        "    words.append(word)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LZJecSzvk1Cd"
      },
      "source": [
        "# Encode the sentence (sentence embedding = mean of word-embeddings)\n",
        "def encode_sentence(text, vocab2index, N=160):\n",
        "    tokenized = tokenize(text)\n",
        "    encoded = np.zeros(N, dtype=int)\n",
        "    enc1 = np.array([vocab2index.get(word, vocab2index[\"UNK\"]) for word in tokenized])\n",
        "    length = min(N, len(enc1))\n",
        "    encoded[:length] = enc1[:length]\n",
        "    return encoded, length"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "qGZfuN3Vk5iL",
        "outputId": "b3110d22-eb33-4936-d3c4-0fdb5b3ea8fb"
      },
      "source": [
        "# we create an encoding for each sentence that contains the index of the first N words of the sentence\n",
        "data_lstm['encoded'] = data_lstm['description'].apply(lambda x: np.array(encode_sentence(x,vocab2index ),dtype=object))\n",
        "data_lstm.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>description</th>\n",
              "      <th>major_code</th>\n",
              "      <th>label</th>\n",
              "      <th>encoded</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Réalise des travaux mécanisés agricoles, sylvi...</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "      <td>[[2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 3, 13, 7...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Tracteur, moissonneuse-batteuse, ensileuse, ép...</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "      <td>[[38, 7, 39, 7, 40, 7, 41, 42, 7, 43, 44, 45, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Depuis la cabine de sa machine, le conducteur ...</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "      <td>[[111, 27, 112, 23, 113, 114, 7, 45, 46, 115, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Réalise des opérations de coupe et d''entretie...</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "      <td>[[2, 3, 186, 23, 116, 34, 187, 17, 11, 16, 7, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Avec ses cordes de rappel, son harnais, ses mo...</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "      <td>[[155, 171, 210, 23, 211, 7, 145, 212, 7, 171,...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                         description  ...                                            encoded\n",
              "0  Réalise des travaux mécanisés agricoles, sylvi...  ...  [[2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 3, 13, 7...\n",
              "1  Tracteur, moissonneuse-batteuse, ensileuse, ép...  ...  [[38, 7, 39, 7, 40, 7, 41, 42, 7, 43, 44, 45, ...\n",
              "2  Depuis la cabine de sa machine, le conducteur ...  ...  [[111, 27, 112, 23, 113, 114, 7, 45, 46, 115, ...\n",
              "3  Réalise des opérations de coupe et d''entretie...  ...  [[2, 3, 186, 23, 116, 34, 187, 17, 11, 16, 7, ...\n",
              "4  Avec ses cordes de rappel, son harnais, ses mo...  ...  [[155, 171, 210, 23, 211, 7, 145, 212, 7, 171,...\n",
              "\n",
              "[5 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 102
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 959
        },
        "id": "zHDBSMwyK_oy",
        "outputId": "0bdd71e2-03f4-42c3-9fa8-160835b629a5"
      },
      "source": [
        "# create train 70 %\n",
        "# create test 30 %\n",
        "X_tr, X_val, y_train, y_valid = train_test_split(data_lstm.index.values, \n",
        "                                                  data_lstm.label.values, \n",
        "                                                  test_size=0.3, \n",
        "                                                  random_state=seed_val, \n",
        "                                                  stratify=data_lstm.label.values)\n",
        "\n",
        "data_lstm['data_type'] = 'not_set'\n",
        "data_lstm.loc[X_tr, 'data_type'] = 'train'\n",
        "data_lstm.loc[X_val, 'data_type'] = 'val'\n",
        "X_train = list(data_lstm[data_lstm.data_type=='train'][\"encoded\"])\n",
        "X_valid = list(data_lstm[data_lstm.data_type=='val'][\"encoded\"])\n",
        "y_valid_desc = list(data_lstm[data_lstm.data_type=='val'][\"description\"])\n",
        "y_train_desc = list(data_lstm[data_lstm.data_type=='train'][\"description\"])\n",
        "y_train = list(data_lstm[data_lstm.data_type=='train'][\"label\"])\n",
        "y_valid = list(data_lstm[data_lstm.data_type=='val'][\"label\"])\n",
        "data_lstm.groupby(['major_code', 'label', 'data_type']).count()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>description</th>\n",
              "      <th>encoded</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>major_code</th>\n",
              "      <th>label</th>\n",
              "      <th>data_type</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">A</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">0</th>\n",
              "      <th>train</th>\n",
              "      <td>52</td>\n",
              "      <td>52</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>22</td>\n",
              "      <td>22</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">B</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">1</th>\n",
              "      <th>train</th>\n",
              "      <td>28</td>\n",
              "      <td>28</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>12</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">C</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">2</th>\n",
              "      <th>train</th>\n",
              "      <td>31</td>\n",
              "      <td>31</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>13</td>\n",
              "      <td>13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">D</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
              "      <th>train</th>\n",
              "      <td>52</td>\n",
              "      <td>52</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>22</td>\n",
              "      <td>22</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">E</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">4</th>\n",
              "      <th>train</th>\n",
              "      <td>46</td>\n",
              "      <td>46</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>20</td>\n",
              "      <td>20</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">F</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">5</th>\n",
              "      <th>train</th>\n",
              "      <td>67</td>\n",
              "      <td>67</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>29</td>\n",
              "      <td>29</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">G</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">6</th>\n",
              "      <th>train</th>\n",
              "      <td>45</td>\n",
              "      <td>45</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>19</td>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">H</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">7</th>\n",
              "      <th>train</th>\n",
              "      <td>151</td>\n",
              "      <td>151</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>65</td>\n",
              "      <td>65</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">I</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">8</th>\n",
              "      <th>train</th>\n",
              "      <td>41</td>\n",
              "      <td>41</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>18</td>\n",
              "      <td>18</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">J</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">9</th>\n",
              "      <th>train</th>\n",
              "      <td>49</td>\n",
              "      <td>49</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>21</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">K</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">10</th>\n",
              "      <th>train</th>\n",
              "      <td>142</td>\n",
              "      <td>142</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>61</td>\n",
              "      <td>61</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">L</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">11</th>\n",
              "      <th>train</th>\n",
              "      <td>38</td>\n",
              "      <td>38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>17</td>\n",
              "      <td>17</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">M</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">12</th>\n",
              "      <th>train</th>\n",
              "      <td>85</td>\n",
              "      <td>85</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>37</td>\n",
              "      <td>37</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">N</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">13</th>\n",
              "      <th>train</th>\n",
              "      <td>45</td>\n",
              "      <td>45</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>19</td>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                            description  encoded\n",
              "major_code label data_type                      \n",
              "A          0     train               52       52\n",
              "                 val                 22       22\n",
              "B          1     train               28       28\n",
              "                 val                 12       12\n",
              "C          2     train               31       31\n",
              "                 val                 13       13\n",
              "D          3     train               52       52\n",
              "                 val                 22       22\n",
              "E          4     train               46       46\n",
              "                 val                 20       20\n",
              "F          5     train               67       67\n",
              "                 val                 29       29\n",
              "G          6     train               45       45\n",
              "                 val                 19       19\n",
              "H          7     train              151      151\n",
              "                 val                 65       65\n",
              "I          8     train               41       41\n",
              "                 val                 18       18\n",
              "J          9     train               49       49\n",
              "                 val                 21       21\n",
              "K          10    train              142      142\n",
              "                 val                 61       61\n",
              "L          11    train               38       38\n",
              "                 val                 17       17\n",
              "M          12    train               85       85\n",
              "                 val                 37       37\n",
              "N          13    train               45       45\n",
              "                 val                 19       19"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 103
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I11HPA-boY9R"
      },
      "source": [
        "# we create a class for a dataset\n",
        "# the __getitem__ returns :\n",
        "## - the word indices of the sentence\n",
        "## - the label of the sentence\n",
        "## - the number of words in the sentence\n",
        "## - the description\n",
        "class DescriptionDataset(Dataset):\n",
        "    def __init__(self, X, Y,Y_desc):\n",
        "        self.X = X\n",
        "        self.y = Y\n",
        "        self.desc = Y_desc\n",
        "        \n",
        "    def __len__(self):\n",
        "        return len(self.y)\n",
        "    \n",
        "    def __getitem__(self, idx):\n",
        "        return torch.from_numpy(self.X[idx][0].astype(np.int32)), self.y[idx], self.X[idx][1],self.desc[idx]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TcTuzjJkkjqV"
      },
      "source": [
        "train_ds = DescriptionDataset(X_train, y_train,y_train_desc)\n",
        "valid_ds = DescriptionDataset(X_valid, y_valid,y_valid_desc)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1fHUKzgj8W4B"
      },
      "source": [
        "batch_size = 5000 #hyperparameter of the model : if too low, computational time is much higher. \n",
        "#but this hyperparameter does not have a high impact on performances\n",
        "#(\"Breuel, Thomas M. \"Benchmarking of LSTM networks.\" arXiv preprint arXiv:1508.02774 (2015)\")\n",
        "vocab_size = len(words)\n",
        "train_dl = DataLoader(train_ds, batch_size=batch_size, shuffle=True)\n",
        "val_dl = DataLoader(valid_ds, batch_size=batch_size)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GzmPfWcM7SSF",
        "outputId": "03757ab3-899e-4918-ab91-f2fbdd577674"
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Thu Apr 15 17:13:45 2021       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 460.67       Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   68C    P0    30W /  70W |   3922MiB / 15109MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gYMwFfOb7SSG",
        "outputId": "40ab38fb-b55a-41eb-b2dc-73ae98c6cced"
      },
      "source": [
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "\n",
        "print(device)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "cuda\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aJqHVpu7QTBH"
      },
      "source": [
        "# General function to train the models\n",
        "def train_model(model, epochs=10, lr=0.001):\n",
        "  model.to(device)\n",
        "  \n",
        "  random.seed(seed_val)\n",
        "  np.random.seed(seed_val)\n",
        "  torch.manual_seed(seed_val)\n",
        "  torch.manual_seed(seed_val)\n",
        "  if torch.cuda.is_available():\n",
        "    torch.cuda.manual_seed_all(seed_val)\n",
        "  parameters = filter(lambda p: p.requires_grad, model.parameters())\n",
        "  optimizer = torch.optim.Adam(parameters, lr=lr)\n",
        "  loss_function = F.cross_entropy\n",
        "  for i in range(epochs):\n",
        "      model.train()\n",
        "      sum_loss = 0.0\n",
        "      total = 0\n",
        "      for x, y, l,d in train_dl:\n",
        "          x = x.long().to(device)\n",
        "          y = y.long().to(device)\n",
        "          y_pred = model(x, l)\n",
        "          optimizer.zero_grad()\n",
        "          loss = loss_function(y_pred, y)\n",
        "          loss.backward()\n",
        "          optimizer.step()\n",
        "          sum_loss += loss.item()*y.shape[0]\n",
        "          total += y.shape[0]\n",
        "      val_loss, val_acc = validation_metrics(model, val_dl)\n",
        "      if i % 20 == 1:\n",
        "          print(\"train loss %.3f, val loss %.3f, val accuracy %.3f\" % (sum_loss/total, val_loss, val_acc))\n",
        "\n",
        "def validation_metrics (model, valid_dl):\n",
        "  model.eval()\n",
        "  correct = 0\n",
        "  total = 0\n",
        "  sum_loss = 0.0\n",
        "  for x, y, l,d in valid_dl:\n",
        "      x = x.long().to(device)\n",
        "      y = y.long().to(device)\n",
        "      y_hat = model(x, l)\n",
        "      loss = F.cross_entropy(y_hat, y)\n",
        "      pred = torch.max(y_hat, 1)[1]\n",
        "      correct += (pred == y).float().sum()\n",
        "      total += y.shape[0]\n",
        "      sum_loss += loss.item()*y.shape[0]\n",
        "  return sum_loss/total, correct/total"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CgBmrcFPGc9p"
      },
      "source": [
        "# Create 3 objects\n",
        "## W : embedding_matrix (1 line = 1 token and columns = embedding dimensions)\n",
        "## vocab : vocabulary\n",
        "## vocab_to_idx : dictionnary containing the link between each token and its id\n",
        "def get_emb_matrix(pretrained, word_counts):\n",
        "    \"\"\" Creates embedding matrix from word vectors\"\"\"\n",
        "    emb_size = pretrained.vector_size\n",
        "    vocab_size = len(word_counts) + 2\n",
        "    vocab_to_idx = {}\n",
        "    vocab = [\"\", \"UNK\"]\n",
        "    W = np.zeros((vocab_size, emb_size), dtype=\"float32\")\n",
        "    W[0] = np.zeros(emb_size, dtype='float32') # adding a vector for padding\n",
        "    W[1] = np.random.uniform(-0.25, 0.25, emb_size) # adding a vector for unknown words \n",
        "    vocab_to_idx[\"UNK\"] = 1\n",
        "    i = 2\n",
        "    for word in word_counts:\n",
        "        if word in pretrained.vocab:\n",
        "            W[i] = pretrained[word]\n",
        "        else:\n",
        "            W[i] = np.random.uniform(-0.25,0.25, emb_size)\n",
        "        vocab_to_idx[word] = i\n",
        "        vocab.append(word)\n",
        "        i += 1   \n",
        "    return W, np.array(vocab), vocab_to_idx"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZDOVxrGVGoSZ"
      },
      "source": [
        "pretrained_weights, vocab, vocab2index = get_emb_matrix(w2vecembedding, counts)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hpuG-cHCHHEE"
      },
      "source": [
        "def matrix_pred (model, valid_dl):\n",
        "  model.eval()\n",
        "  data={\"description\":[],\"label\":[],\"y_pred\":[]}\n",
        "  y_hat = np.array([], dtype=np.int32).reshape(0,len(tag_to_ix))\n",
        "  for x, y, l, d in valid_dl:\n",
        "    x = x.long().to(device)\n",
        "    y = y.long().to(device)\n",
        "    y_hat_tmp = model(x, l).cpu()\n",
        "    y_hat = np.concatenate(( y_hat, y_hat_tmp.detach().numpy()))\n",
        "    pred = torch.max(y_hat_tmp, 1)[1]\n",
        "\n",
        "    data[\"label\"] += [key_list[val_list.index(i)] for i in y]\n",
        "    data['y_pred'] += [key_list[val_list.index(i)] for i in pred]\n",
        "    data['description'] += d\n",
        "  return pd.DataFrame(data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7qIEWspUHNo-"
      },
      "source": [
        "def f1_w_score(model,valid_dl):\n",
        "  data = matrix_pred(model,valid_dl)\n",
        "  y_true = data.label\n",
        "  y_pred = data.y_pred\n",
        "  return f1_score(y_true, y_pred, average='weighted')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cLmEnCtyHevP"
      },
      "source": [
        "def diagnostics(model, valid_dl):\n",
        "  data = matrix_pred(model,valid_dl)\n",
        "  y_true = data.label\n",
        "  y_pred = data.y_pred\n",
        "  print(classification_report(y_true, y_pred) )\n",
        "  cm = confusion_matrix(y_true, y_pred , labels = key_list) \n",
        "  df_cm = pd.DataFrame(cm, index=key_list, columns=key_list)\n",
        "  # config plot sizes\n",
        "  sn.set(font_scale=1.2)\n",
        "  sn.heatmap(df_cm, annot=True, annot_kws={\"size\": 18}, cmap='coolwarm', linewidth=0.5, fmt=\"\")\n",
        "  plt.tick_params(axis='both', which='major', labelsize=10, labelbottom = False, bottom=False, top = False, labeltop=True)\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vNIuKNuKHfzj"
      },
      "source": [
        "import os, sys\n",
        "# To hind the print\n",
        "class HiddenPrints:\n",
        "    def __enter__(self):\n",
        "        self._original_stdout = sys.stdout\n",
        "        sys.stdout = open(os.devnull, 'w')\n",
        "\n",
        "    def __exit__(self, exc_type, exc_val, exc_tb):\n",
        "        sys.stdout.close()\n",
        "        sys.stdout = self._original_stdout\n",
        "# Function to test a parameter on lstm models (models 2 and 3)\n",
        "def test_parameter_lstm(model_class, embedding_dim, hidden_dim, lr = 0.01, epochs = 400):\n",
        "  model = model_class(vocab_size, embedding_dim = embedding_dim,\n",
        "                      hidden_dim = hidden_dim)\n",
        "  with HiddenPrints():\n",
        "    train_model(model, epochs=epochs, lr=lr)\n",
        "  return f1_w_score(model, val_dl)\n",
        "# Function to test a parameter on lstm models with word2vec (models 4 to 7)\n",
        "def test_parameter_lstm_w2vec(model_class, embedding_dim, hidden_dim,vec_weights, freeze_emd, lr = 0.01, epochs = 400):\n",
        "  model = model_class(vocab_size, embedding_dim = embedding_dim,\n",
        "                      hidden_dim = hidden_dim,\n",
        "                      vec_weights = vec_weights,\n",
        "                      freeze_emd = freeze_emd)\n",
        "  with HiddenPrints():\n",
        "    train_model(model, epochs=epochs, lr=lr)\n",
        "  return f1_w_score(model, val_dl)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nWTb3xHNIFTe"
      },
      "source": [
        "## **Model 2** : LSTM (fixed-length inputs + random initialization of word-embeddings)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ex145gOjkipw"
      },
      "source": [
        "class LSTM_fixed_len(torch.nn.Module) :\n",
        "  def __init__(self, vocab_size, embedding_dim, hidden_dim) :\n",
        "      super().__init__()\n",
        "      self.embeddings = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)\n",
        "      self.lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first=True)\n",
        "      self.linear = nn.Linear(hidden_dim, len(tag_to_ix))\n",
        "      self.dropout = nn.Dropout(0.2)\n",
        "      \n",
        "  def forward(self, x, l):\n",
        "      x = self.embeddings(x)\n",
        "      x = self.dropout(x)\n",
        "      lstm_out, (ht, ct) = self.lstm(x)\n",
        "      return self.linear(ht[-1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "acWYN3U4NcWY",
        "outputId": "bd756d5e-28bf-430d-db10-6cf3e8c1c0b1"
      },
      "source": [
        "model_fixed =  LSTM_fixed_len(vocab_size, embedding_dim = 50,\n",
        "                              hidden_dim = 50)\n",
        "train_model(model_fixed, epochs=400, lr=0.01)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train loss 2.580, val loss 2.571, val accuracy 0.149\n",
            "train loss 1.924, val loss 2.717, val accuracy 0.155\n",
            "train loss 1.481, val loss 3.062, val accuracy 0.147\n",
            "train loss 1.341, val loss 3.361, val accuracy 0.141\n",
            "train loss 1.263, val loss 3.657, val accuracy 0.136\n",
            "train loss 1.238, val loss 3.748, val accuracy 0.139\n",
            "train loss 1.180, val loss 3.680, val accuracy 0.149\n",
            "train loss 1.287, val loss 3.887, val accuracy 0.141\n",
            "train loss 1.272, val loss 3.785, val accuracy 0.147\n",
            "train loss 1.251, val loss 3.878, val accuracy 0.149\n",
            "train loss 1.206, val loss 4.013, val accuracy 0.152\n",
            "train loss 1.241, val loss 3.890, val accuracy 0.123\n",
            "train loss 1.206, val loss 3.947, val accuracy 0.123\n",
            "train loss 1.167, val loss 3.987, val accuracy 0.120\n",
            "train loss 1.156, val loss 4.014, val accuracy 0.155\n",
            "train loss 1.119, val loss 4.006, val accuracy 0.147\n",
            "train loss 1.167, val loss 4.052, val accuracy 0.141\n",
            "train loss 1.104, val loss 4.168, val accuracy 0.139\n",
            "train loss 1.058, val loss 4.290, val accuracy 0.141\n",
            "train loss 1.183, val loss 4.695, val accuracy 0.152\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 664
        },
        "id": "5jZK4jAiNP7m",
        "outputId": "35e8d9ad-186d-4be6-b7d7-d794fdba2de1"
      },
      "source": [
        "print(\"***Model 2 : LSTM (fixed-length inputs + random initialization of word-embeddings)***\")\n",
        "pred_vs_true_m2 = matrix_pred(model_fixed, val_dl)\n",
        "diagnostics(model_fixed, val_dl)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "***Model 2 : LSTM (fixed-length inputs + random initialization of word-embeddings)***\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           A       0.00      0.00      0.00        22\n",
            "           B       0.00      0.00      0.00        12\n",
            "           C       0.00      0.00      0.00        13\n",
            "           D       0.00      0.00      0.00        22\n",
            "           E       0.00      0.00      0.00        20\n",
            "           F       0.09      0.03      0.05        29\n",
            "           G       0.10      0.05      0.07        19\n",
            "           H       0.17      0.29      0.22        65\n",
            "           I       0.00      0.00      0.00        18\n",
            "           J       0.00      0.00      0.00        21\n",
            "           K       0.16      0.25      0.19        61\n",
            "           L       0.00      0.00      0.00        17\n",
            "           M       0.17      0.19      0.18        37\n",
            "           N       0.08      0.05      0.06        19\n",
            "\n",
            "    accuracy                           0.12       375\n",
            "   macro avg       0.06      0.06      0.06       375\n",
            "weighted avg       0.09      0.12      0.10       375\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEDCAYAAAAsr19QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd1gU1/7G360ssCxFRQERlSCIiAISsUexN0RsMbEkkhsTS4opP42J0eTmmug1uVGTqJhYQiSxxAK2YMSSWBCNBaUYK1aUtgtsYXd/f2x2ZdnCzO5ZdpD5PM88Dzsz57vvOXPmcPbMOe9wtFqtFiwsLCwsjQ6uswWwsLCwsNgG24CzsLCwNFLYBpyFhYWlkcI24CwsLCyNFLYBZ2FhYWmksA04CwsLSyOFUQ14ZmYmQkND8ffff9sVp2PHjkhISMDo0aORmJiIs2fP2hWvuLgYb731FgYOHIixY8filVdewfXr123WNWLECIwePRrff/89NBqNzbr08fTb2rVricUqKiqyOdajR48wb948xMfHY+zYsZg4cSJ+++032nGioqKMPu/YsQNLliyxWZeluM6OVzv9kSNHMGTIENy5c8dpevSEhobinXfeMXyuqalBXFwcXn31VZtiLV261PB5/fr1WLlyJRGdTRm+swXUJj09HTExMcjIyMDcuXNtjiMSibBr1y4AwLFjx7BixQr8+OOPNsXSarWYPXs2xowZgy+//BIAkJeXh8ePH6Ndu3Y263r8+DHmzZsHmUxmc15rx7MXUrG0Wi1mzZqFMWPG4L///S8A4M6dO/j999/tjv20c+LECXz66adYv349AgICnC0Hbm5uKCwshFwuh0gkwh9//IGWLVvaFEsoFOLgwYP417/+BR8fH8JKmy6M6YFXVlYiJycH//73v5GRkUEsrkwmg0QisTn9yZMnwefz8fzzzxv2hYWFoVu3bnbpatasGT755BOkpqbiaVpLdfLkSQgEAqPyCggIwJQpU5yoivlkZ2dj4cKF+O6779CmTRtnyzHQr18/ZGVlAQAyMjIwYsQIm+Lw+XxMnDgRGzduJKiOhTE98EOHDqFPnz5o164dvL29cenSJURERNgUSy6XIyEhAQqFAsXFxXZVmsLCQnTq1Mnm9NYIDAyEWq3G48eP0bx5c9rp9fnU8+qrr2L48OE2aakdq3Xr1li9erVNcQoLCxEeHm5TWmuaAKC8vBwDBgwgEptJKJVKzJo1C5s2bUJwcLCz5RgxfPhwfPPNN+jfvz/y8/ORlJSEnJwcm2K98MILGD16NJKTkwmrbLowpgHPyMjA1KlTAegqTUZGhs0NeO3hgHPnzuH9999Heno6OBwOMb1MgIlDKHVZvHgxcnJyIBAIsH37drs07dixA5cuXSIt0enw+XxERUVh27ZtWLhwobPlGBEWFoaioiKkp6ejX79+dsUSi8VISEjApk2bIBKJCCls2jBiCKWsrAwnT57EwoULMWDAAKxfvx779u0jMrQQFRWF0tJSlJSU2JQ+JCQEubm5duswx+3bt8Hj8dCsWTOHxHcGISEhuHz5suHzokWLsGHDBpSWljpRFbPhcrn46quvcPHiRXz33XfOlmPCgAED8MUXX9g8fFKbadOmYfv27aiuriagjIURDfiBAweQkJCAw4cP4/fff8eRI0fQunVrnDlzxu7Yf//9N9RqNby8vGxKHxcXB6VSiZ9//tmwLy8vz25tJSUlWLRoEV544YWn6pdBXFwcFAoFfvrpJ8M+uVzuREWNA1dXV6xZswZ79uzB1q1bnS3HiHHjxmHWrFkIDQ21O5aXlxeGDh2Kbdu2EVDGwoghlPT0dLzyyitG+wYPHoz09HTExsbSjld77FSr1eLzzz8Hj8ezSRuHw8GqVavw2WefYd26dXBxcUFAQAAWLFhgs66amhrweDwkJCTgpZdesklX7Xh6+vTpYzTtyxlwOBysXr0a//nPf5CSkgIfHx+4uro6XZejqKmpgVAoJBLLy8sLKSkpeOGFF+Dj44P4+Hib4lRXV6Nv376Gzy+99JJd9axVq1aG4U0SvPzyy0hNTSUWrynDYe1kWVhsJy8vDwsXLmR7lCxOgRE9cBaWxsiWLVuwefNmm36NsbCQgO2Bs7CwsDRSGPEQk4WFhYWFPmwDzsLCwtJIYRtwFhYWlkYK24CzsLCwNFLYBpyFhYWlkcI24CwsLCw0WLZsGUaMGIHo6Gj07t0bCxYsMLGKuHz5MiZNmoQuXbrgueeew6ZNm6zG1Gg0WLFiBXr27ImoqCjMmDGDkic824CzsLCw0IDH42HZsmU4deoUdu3ahfv372P+/PmG4zKZDMnJyejduzdOnz6Nr776CqtWrcL+/fstxkxJSUF6ejp+/PFHHD9+HP7+/pg5c2a9L3xh54GzsLA0eSoqKlBRUWGyXyKR1Ps+gcOHD2PevHmGN3/t2LEDK1aswNGjR8Hl6vrIy5Ytw8WLFy32xAcMGIDk5GRMnjzZoKdnz5744YcfrNqJsCsxWRjJypUrsWrVKovH+Xy+w1wiWZ4OMgTUzbeurZhttr7Nnj0bc+bMsZr2xIkTCAsLM3zOy8tDeHi4ofEGgIiICIsmZVKpFHfu3DGyz5ZIJAgKCsKVK1eY04D3HnXE7P43XgnG+NGtceTPYpzMKUFQoDvGjfTH+dxyvPnhBdT9jXB8Tz+L8ZgQa9byMqP9oW34GBLnght31SiVaaDRAP7NeegRIUS1Uov/bJJCVmUcbPU7XsR06bWR0KXXRvJa7sxWm8Th+8dj4szWJvvv3y7AkYzvEdr1OZN0Y2J5jK5jCzcozWqrzdBuPMSGcuEi4ODkFTXSTxnn8dPpOuMsktdy8v8Zv/900lAJBvUQ4+xlOQpvK6BWA+HtXdCjixtu3lPio9UPoaox1f7T0tZE62tDMm3aNCQmJprsr6/3vXfvXmzdutXolY0ymQweHh4mcWQymdkY+v11v8vDw8NiGj1O74G3a+OGpJEByPqzGAv/88RH+t6Darz1aggG9vXFb0ceNupY+bdqkH/LtMZfLapB8mh3xHUSIjNb0ah1kdbm1yYUfm1Me1Db1y8CAMT2S3KKLpKx6uLnw0GPcC4OnFFj+LP0bk2S1/LUxWrsOixFteJJy3roVCXuP65B4gAJ+se64+CJSkqxHFle9cERULdppjJUUpeMjAx8/PHH+Pbbb43e2iUWi/H48WOjcysqKiAWi83G0e+XSqVG+6VSqcU0epz+EHNgX19wuRz8ssv4ieueA/dQLVdj8HO+jT6WJUoqdA8o3ETUKxpTdTWENqW8CudP7oOnTyuEduntFF2OyiOHA4zpyUPhHS0u37L+4IoOtlzL63dURo23npPnqwAArVsKKMdqiPpqCZ4rj/JGl61bt2Lx4sX47rvvEBcXZ3QsLCwMly9fNnoAmZubazTMUhsPDw8EBAQYvW1KKpXi1q1b6Nixo1UdTm/Aw0I8oFZrcaXA+AGCUqVF4TUZwkI8LKRsPLH08HmAuysHXmIOwoL4eH6QKwAg95qq0etylLbaXDh9AIpqGWL6jAGXS/2mY3q9AICe4Vy08OQg/ZSZsQkakLqW5vDx1P0qKJdR/wfj6DphDS6fQ3mjw6ZNm7B8+XKsX78eMTExJscHDx4MtVqNb7/9FkqlEhcuXMDWrVuNXvRdl0mTJmH9+vW4fv06qqqqsGzZMrRt29Zs/NrQHkI5c+YMMjIysGjRIrpJzdLcR4jyChVUNab/8R+VKBAZ7gk+n4MaM8cbSyw9vSKFmBDv9iROuRobMirx9x3TMeDGpstR2mqTnbUdHA4Hsf3GOk2XI/LoLQbiu/Jw+LwaZTLAy/qvZquQupZ14XCAxHgP1Ki1+POvKsrpHF0nrEFnCIUO//73v8Hn801ecpGRkQF/f3+IxWKkpKRg8eLFWLNmDby9vTFr1iwMGzbMcG5ycjL8/f2xZMkSw2epVIrJkyejuroaMTEx+Pbbb40ehJqDUgN++fJl7NmzBwcOHEBAQAAGDx5MN88WEbnwoKox/x9dqdT8cw4Xspr6KyBTY+k5X6jC/ccyuAg5CPTloXMwH+6u9CoZU3U5Spue4rvXcaPgLJ7pFAcfX9OHmw2lyxF5HN2DjxKZFn/k2j90Qupa1mXqKC90CHJB2v5y3HtE/VeCI+tEfdDtWVMlPz+/3nPCw8ONXsNYl5SUFKPPXC4X8+bNw7x582hpsdiAX79+HRkZGUhPT4e3tzeGDx8OrVaLzZs30/qC+pAr1PAWmX8llVDI/eccahWbqbH0lMm0KJPpKv+FqyqcK+DivRc9IORzcPA0tQdMTNXlKG16so/o3mgf+9w42mmZXC+6tOci2J+D9ftqoCHQCSV1LWszfpAEQ3qKceiUDLuzpPUnqIUj60R9OKoHziQs9s+HDRuGkydPYs2aNdiyZQumTJlSb3feFh6VKOEpEUBg5r9lcx8XlJYrKf+8YmosS9x9pEHRQzX6dnWhnIapuhypTa2uQc7xXXATeyGi20Da6ZlaL3hcYFgsDwVFWkirtfDxAHw8AC93XWyRUPfZQvtHCVuvpZ6kgRIkxkuQdaYS638tqz9BHRqivlrCUWPgTMJii7xq1Sq0aNECU6dOxcKFC3HixAk4YtFmXqEUPB4HHTsYT+ERCjgIaS9G/lXr8yAbQyxrCPgcuNH4ictUXY7UduVsFmTljxHVaxT4AvqtGVPrhYAPiF05CAvk4u0koWFLHqab5dE1mIe3k4SICbGv42TLtQR0jXfSQAmO5FRi3fbS+hOYoaHqqzl4Qi7lrbFiUfnAgQPx5ZdfYt++fejevTs2btyIkpISLFq0CMePHycm4NCxYmg0WkxICDDaP2qIH1xFPBzMetDoY0nczN88IYF8+Dfn4sZd6mOKTNVFWlttngyfUJ/77ShdJGMpVcCWwyqTbfcJXbkXFGmw5bAKebfrH2IgfS0T4z2QNFCCY2crsXZbqdnFNlRwVJ2gAofLobw1Vup9iOnm5oZRo0Zh1KhRKC8vx/79+7Fu3Tr07k19Hq41rt2sxI6Muxg3KgD/nh+OEzklaNvaDeNGBeDcxTJak/yZGmviIFd4unNRcKsGJRUa8PlAm5Z8xIQJIFcCO7LkjV4XaW16KkofouDCcQQGd4ZfYAfa6UnrIhlLowVyb5q2jF5i3b4SqdbscXOQvJaD4twxfpAniktrcOmqAj27uBkdL5epcekqtfF0R9QJqnB4jbdnTRVa0wg9PT0xceJETJw4kaiIr1Ou4v5DOUYP8UOP2GYor1BhW/pdrE+9Tvs/PxNj5eSp8Gy4EM+GCyF240Cr1S2wOH5eicxsOUql9IQxVRdJbXrOHP0VGo3apoeXjtJFOo8kIHkt2wfqhqlaePPx2gQfk+OXrylw6Wox5XjOKi8ur/H2rKni9KX0AKDRAGk7i5C2s6j+kxthrLP5KpzNt38hhR6m6gLIlj8ADEh4FQMSXrU7DhPrhSXKZNR8U2pD8lqu2VqKNVttG/M2h6PLyxKNeWiEKoxowFlYWFhI05gfTlKF9QNnYWF5Ksnp34vyuTGH/3CgEsfB9sBZGE1ZWRnWrFmDzMxM3L9/H+7u7ggJCcEbb7yBbt26OVseC4Nhh1AIw2SvZqbGGv7yRZNYe7/vjApZDf6+WY1nglxRJdfgpfesL+/d+31nol7NJK/lD4fNay5/fAc/rZgCpaIKXXqNQ6d+baGolqH4Tj5+/v0BLtZZFPhSf2bXMab6xJOsYyRj2Qv7ELMBYKpXM1NjAcDL7+fhfrHugdU3S0LgKrJtrI+0LtLx9vzwLjQaNWZ8uBtiT9ttR5l6LZnqEw+Qq2OkY9GhKfTAnT7Kz1SvZqbGAmC4GeyFtC6S8W4VZqPoag66D06G2NMXarUKKmU1LT2O0MVUP3am1jHSsejA4XIpb40Vp/fAmerVzNRYJCGti2S8a5d0QwQSbz9sXT0T13KPQqtRw9u3LXqNeB0R3ROcostRfuwuQg4EPKBVMx7G9BUBcL5PfGOHJ2i8DTNVnJ7D+vyCvT2F4FM0m2kKsUhCWhfJeI/vXwcA7E/9EPLKMoycthTDp34GHl+A9B/ew4U/tztFlyOuZa9IIb6Y5Yl/z/TEnPFiuIo4xH3inVXHnAm7lN4MJSUl8Pb2BodDJtNM9WpmaiySkNZFMp5SoXvnotDFHZPf3gQeX7c6sEOXgfj2w4E4svNLdI5LpPTzl+nXkqk+8Y2dxjw0QhWrOfzrr78wZcoUzJ49G5cvX8bIkSMxatQo9OzZE0ePHiUiQK5QQ8A3L8MWr+anPRZJSOsiGY8v0A0jhMeOMDTeACBy90RI5ABUVhTj8YPrDa7LEdeyTKZF/q0aXLiqQsafcmzaV4UxfV0x+FnqFrBMrWPOpCn0wK024EuWLMGrr76KESNGYNq0afj000/xxx9/4Mcff8SKFSuICGCqVzNTY5GEtC6S8STeLQEA7pIWJsfEnrp98qryBtfFVD92ptYxZ+KoBjwjIwOTJ09GdHQ0QkNDjY7t3r0bUVFRRlt4eDhGjx5tMd6OHTsQFhZmlGbSpEmUtFhtwNVqNXr37o1hw4ahefPm6Nq1KwAgODiYUnAqMNWrmamxSEJaF8l4fm0jAQDSsvsmx6Slun3uHs0aXBdT/diZWsecCZfPo7zRQSKRYPLkyViwYIHJsdGjR+PcuXOG7fTp0/D29kZCgvWH7v7+/kbp0tLSqOXR6sFaY0gikcjoGKkxcKZ6NTM1FklI6yIZr0OXgRCK3JF7ajeU8krDfln5QxScPwSflm3h7RvU4LqY6sfO1DrmTLg8DuWNDn369MHIkSMRGBhY77kHDx6ETCZDUpJtXvb1YfUhZl5eHqKjo6HVaqFQKBAdHQ0A0Gq1UCrpuaVZgqlezUyNBQADenjBt5nurS2eHjzw+VxMGqkbVnj4WIXfT1B79RVpXSTjidw9MSDpfexP/QibvpiIyJ5JUNeocO7oFqhrVBg08UOn6GKqHztT6xjpWHSgMzRSUVGBiooKk/0SiQQSicRMCmr89NNPGD58OLy8vKye9/DhQ8M7FiIiIvDmm28iLCys3vhWG/ArV67QkGo7TPVqZmqswX28ERkmNto3dWwrAMCFPBmtG4K0VzPJeF37TISr2BunDqbg2O7/gcPhwL99FEa/vBytn4lxmi6m+rEztY6RjEUHOrNQNm7ciFWrVpnsnz17NubMmWPT9xcUFODMmTN4//33rZ4XGxuL3bt3IygoCFKpFGvXrsXUqVOxZ88etGzZ0mpapy/kAZjr1czUWP/3BbXZF1Qg7dVMOl5o1GCERg22Ow4TryVTfeIBsnWMZCw60OmBT5s2DYmJiSb77e19R0REIDIy0up5tYdivLy88N577yEzMxNZWVn1vjyHEQ04CwsLC2noNOD2DpXURSaTYffu3fjggw9sSs/hcCi9RJ71A2dhYXkquf069QeHgd9QX9mrVqtRU1OD7OxszJgxAxcuXAAACAQCw8SP1NRUfP311zhy5IjJBJC6ZGZmonPnzvD19YVMJsO6deuwZcsW7Nq1C/7+/lbTNmgP3JzVJABMGN4CwUGueKatK/xaCPHgkdKq3aTeatKcpSaHA4wfHYCEof5o5StCWbkSh48XIyX1htmFDNZsQ5+WWPp4jo5laz7r2qwCgK83F8N7iBDYkgdPMRc8rm58OPe6CpnZClRUmvY7Vr/jRVQXYL7O2lpfqbwmTcAD5owRwMeDg5NX1Eg/Zbxy8tPpukVNdcvMlvICdGVGIo/6fJKsY/biqJWYu3btwvz58w2f9UMkmzZtQvfu3QEAaWlpSExMNNt4f/TRR7h79y5SUlIAAMeOHcPHH38MqVQKsViMiIgIbNiwod7GG2DIEMr0ca0MfsFiV/sKfW7yE0/ktF9v6zyRRwUgpL3YoicyG4tMLJLxvD24kIg5OF+oMvLK7h3pgpgwoVmv7IbKJ8n6Wpf4KB7crXfYzEK6vJh6T9KC0FTnuowdOxZjx461es6ePXssHluyZInR58WLF2Px4sU2aWFEA85Ef2s2lnP9wJuKV3Zt/Hw46BHOxYEzagx/lt6tSbK8AGbek3RpzEvkqcIItxcm+luzsZzrB26Jp80rWw+HA4zpyUPhHS0u3yLnWWJLeQHMvCfp0uT9wG/evIlHjx4hJsZ4zm1OTg5atGiBNm3aOFQcXZjq+9wUYjkiHtB0vLJ7hnPRwpODLVn2NZwkyoskziz7Jt8D/+yzzyAWi032i8VifPbZZw4TZStM9X1uCrEcEQ9oGl7Z3mIgvisPh8+rUWanZQmJ8iKJM8veUV4oTMJqD/zRo0cmblsAEBoaijt37phJ4VyY6vvcFGI5Ih7QNLyyR/fgo0SmxR+59g+dkCgvkjiz7JtCD9xqAy6VSi0ek8up+zQ0FHKFGt4iodljtvg+s7Ho+4GTjAfovLLLZLqHcxeuqnCugIv3XvSAkM/BwdPUHso5QhcpurTnItifg/X7aqAhMBuDRHmRxKll34jHtqliNYcRERH45ZdfTPZv3boVnTp1cpgoW2Gq73NTiOWIeOZ4mryyeVxgWCwPBUVaSKu18PEAfDwAL3edTpFQ99lC+0cJW8qLJM4sew6HQ3lrrFhtwBcsWIAdO3ZgypQpWLp0KZYuXYoXX3wR27Zts3mJqCNhqu9zU4jliHiWeFq8sgV8QOzKQVggF28nCQ1b8jCdc1/XYB7eThIiJsS+niTd8iKJM8u+KcxCsaq8efPmSEtLw6xZsxAQEICAgADMmjULP//8M1q0MH1TirNhqu9zU4hFOl5T8MpWqoAth1Um2+4TurwVFGmw5bAKebfrH2IgWV4kcWbZc/g8yltjhdJqgbi4OMTFxTlMBBP9rdlYzvUDbwpe2RotkHvTdPjAS6zbVyLVmj1uDpLlBTDznqRLk3+I2VAw1d+ajUUvFsl4TcUrmxSky4up9yQdOJzGOzRCFUY04Ez1t2ZjOS9eU/HKNkeZjJrxVW1IlxdT70laNIEeOGsny8LC8lRS9vlsyud6vW/6Np7GACN64Cws5qisrMTmzZuRkZGBoqIiCIVCtGvXDhMmTEBiYmKjnv4FACtXrjT7Gi89fD4fubm5Dajo6YIdAyeMJT/wvd93NlhXPhPkiiq5xmY/8DdeeWJdeTKnRGddOdIf53PLzVpXWvPdflpi6eORjGXuWtK9jvo0R3KrTPZrNBos/zAZf+efR4/nRuHZAROgVMhx+vh+zJ8/H1mn8pE09Q2jNP06uVn0A7e1/EnkU19fd2Ybrzbk+8dj4szWJuffv12AIxnfI7TrcyZpxsTqZkyQvJZUhmuGduMhNpQLF4F5n3JA51VOUpe9cHiNd3YJVRjRA2eidSUby7k2q9cLL+LqlXOIH/kCJr78jmH/c0Mn4KO5iTh6cJtJA24JpubTr00o/NqYWlVsX78IABDbj/obZRxp22qPza0z7WSb/ErMhoKJ1pVsLOfarMqrKgEAXj7G6w34AgHEHl5wEblSjsXkfNZFKa/C+ZP74OnTCqFdelNO5yjbVnttbp1qJ9sEVmIyogdOCqZarTaFWKRpGxIBN3cPHNi5Ac18/dEuJAJKhRwnsvbg5rUrePFV6iuBmZzPulw4fQCKahl6DX4RXC71IQBH5dFem1unlr2DeuAZGRlITU1FXl4eKisrkZ9vPHwWGhoKFxcX8GoN4aSlpZk1BtSzYcMGbNiwAaWlpejYsSM+/vhjhIWF1avlqWrA67OujAz3BJ/PoeS9wMaiF4s07mIJZs3/Cpu+WYK1y98z7Be5umPmu8sR1b0/5VhMzmddsrO2g8PhILaf9Vd21cUReaxrc+tl6iztFF1UcdRDTIlEgsmTJ0Mul1u0FFm3bp3h/Zj1kZGRgW+++QZr165Fx44dsW7dOiQnJ2P//v1m7bxrQ/lfVElJCUpKSqie7hSoWleyscjHcgQuIjf4twnG4ISpeO29/2Lq6x+hRatApHy5AJf/Okk5DtPzqaf47nXcKDiL4PDu8PE1fbhpDUfkkYTNrVPLnsOlvtGgT58+GDlyJAIDA4nITEtLw/jx49G1a1e4uLjg9ddfB6B7W319WFWu1WqxcuVKdO/eHUOHDsWQIUMQFxdndeqTM5Er1BDwzWfJFqtVNpbzbFaLbhbi8wXTEd4lDuOmvYWouAHoPTAR7332PTy9m2Hzt0ugUVPzkGZyPmuTfWQ7ACD2uXG005LOo97mds8JtV02t84sew6PR3mrqKhAUVGRyVZRUVH/F5lh3rx56N69OxITE806utYmLy8PERERhs9cLhfh4eG4cuVKvd9jdQhlw4YNOHv2LLZt22b4b3P79m18/PHH2LBhA6ZPn04hKw3HoxIl2ga6Q8DnmPxks8VqlY3lHJtVAMjckwqVUoGYnoOM9ru4uKJzdG8c3vczHhXfhW+r+ntBTM6nHrW6BjnHd8FN7IWIbgNppyeZR3M2t8ATwyy9zW2VApDXMwPRqWVPYwhl48aNZjums2fPxpw5c2h97YYNGxAVFQUul4uTJ0/inXfeQU1NDSZPnmz2fJlMBonE2K3Rw8MDMln9To1We+C7du3Cf//7X6OfCoGBgVi2bBl27txJJS8NClOtVptCLNKUleimlmnUpr0zjUb9zzFqPXAm51PPlbNZkJU/RlSvUeAL6BuAk8wjSZvbxmInO23aNBw6dMhkmzZtGu3v7dGjB0QiEYRCIfr27Yvp06dj9+7dFs8Xi8UmL8+RSqX1jn8D9TTgNTU18PHxMdnv4+ODmhrn2FNag6lWq00hFmn8WrcHAPx52LjiV1VK8dfpLLiJJZR63wCz86nnyfAJ9bnftSGZR5I2t04tew6H8iaRSNC6dWuTrW7P2Ba4XC6sOZaEhYXh0qVLhs8ajQaXL19Gx44d641tdQhFIBDYdIwuTLSuZGM5z2YVAAaOnIyTWen49cevcefWVTwT1gWV0gocy9yB8tJHmPzKfHAprrRjcj4BoKL0IQouHEdgcGf4BXaglVYPyTyStLl1pp2so6YRqtVq1NTUQKXSTa1UKHSvqhMIBLhy5Qo0Gg1CQ0PB5XJx6tQp/PDDD5g1a5bFeJMmTcKSJUswePBghIWFYd26dQCAgQPrH0qz2hN85ocAACAASURBVIDn5eUhOjraZL9Wq4VSSc8tzRpMta5kY9GLRfI6NvP1x/wvNiPjl7W4cuE0so8fgFDogsB2HTB++tuIjounpY2p+QSAM0d/hUajtunhZW2cZdvKVF2OWkq/a9cuzJ8/3/A5MjISALBp0yZUVlZi2bJluH//Png8Hvz9/fHmm2/i+eefN5z/0Ucf4e7du0hJSQEAjBgxAsXFxZgzZw5KS0sRHh6OlJQUSkMoDepGaMkLhS7WvFDoYs1z5GmJpY9HMhbJa2nOC8UWrHmh0MWaFwpdLHmh2II1LxRboOqFQgVLXii2QMILpfrHzyif6/riAru/zxk8VQt5WFhYWAw0ATdC1g+chYXlqUS+5XPK54qef9+BShwHI+xkJwxvgeAgVzzT1hV+LYR48EhJyZ7TXDxbY5n76cfhAONHByBhqD9a+YpQVq7E4ePFSEm9YXbxgbWf3c7Updfm6Fi2xDu+px9mLTcdM/b15mJ4DxECW/LgKeaCx9W9Iiz3ugqZ2QpUVJr2O1a/40VUF0D2Wq7YZap5zXvm/S74QjfM+PSsyf63EzjEdOm1/XDY/LHqyjKc2LcGBeczIS29D6HIHS38Q9Bn1BsIDOlmcv5L/cnqspsm0ANnxBDK9HGtDP7KYlf7nhyTjDU3+YmPcdqvt3U+xqMCENJebNHHuLHpIhmLZDxvDy4kYg7OF6pQKtNAowH8m/PQO9IFMWFC/GeTFLIq6uKYei0BoFW7bujYfYLRPi6X/q1JUlf54zv4acUUKBVV6NJrHLx920JRLUPxnXxIy+hN/SNdXpRh34nZMJD0kWaitzhTdZHOI8l4+bdqkH/LdK3B1aIaJI92R1wnITKzFQ2uCyBbXwFA4tMaHaJH2xWDtK49P7wLjUaNGR/uhtjTPstX0uVFmSbwQgdG/Isi6a/MRG9xpuoinceG8H4uqdANd7iJqP88Zuq1rI26RgmVotKuGKR03SrMRtHVHHQfnAyxpy/UahVUymqn66KNg8ysmAQjeuBMhKke0kz2A3dEmfF5gIuQAwEPaNWMhzF9RQCA3GvUGwWmXks91y4eROG5PdBq1BC5+yC4yzDEDnkTLq7O0XXtku45gsTbD1tXz8S13KPQatTw9m2LXiNeR0T3BKfoog07Bt50YaqHNJP9wB1RZr0ihZgQ7/YkTrkaGzIq8fcd6nOqmXotAcA3MBLtI4dA0iwIKoUMt/KOIvfPVNy7lo0xs7ZA4OLe4Joe378OANif+iG8WwRh5LSlUKtVOJ35A9J/eA8adQ0ie9q25L9BacQ9a6pYzaF+SScA7Nu3z+jYihUrHKOIITDVQ5rJfuCOKLPzhSp8/YsMa3ZWYu+fclTLtXB3pdezYuq1BIDEOb+gS78ZaBcxEB1ixmDgCysQO/RNlNwvwMXjm52iSfnPUI7QxR2T396ETt1HI7JnEl6clwoXNwmO7PwSWo3z7XfrhYYXSmPFaq3du3ev4e+1a9caHTt27JhjFDEEpnpIM9kP3BFlVibTIv9WDS5cVSHjTzk27avCmL6uGPysi1N1OZIu/WaAyxPgVl6WU76fL9ANU4XHjgCP/8QZUeTuiZDIAaisKMbjB9edoo0WXC71rZFS7wsdzP1t7vPTxqMSJTwlAgj4pv+dnekhTVIX6Tw2RJndfaRB0UM1+nal3oAz9VpagscTwF3iC3llqVO+X+LdEgDgLmlhckzsqdsnrypvUE02weVR3xopVhvw2m9rrvvm5sb8JmcqMNVDmsl+4A1VZgI+B240hlGYei0tUaNSoLL8AVzFzZ3y/X5tdeZM0rL7Jsekpbp97h7NGlSTTTT1HrjejTAqKgr5+fmIjo42fC4oKGgojU6BqR7STPYDJxlP//aXuoQE8uHfnIsbd6n70TP1WlrqYWcf+B80mhoEhVN/cTNJOnQZCKHIHbmndkMpfzK1UVb+EAXnD8GnZVt4+wY5RRstmsAYuNVZKFTeyUYCkv7KTPQWZ6ou0nkkGW/iIFd4unNRcKsGJRUa8PlAm5Z8xIQJIFcCO7LkTssnqWt59tB3eHDrL/gHd4eHlz9UykrcyjuKu3+fgm+bLojo9aJTdIncPTEg6X3sT/0Im76YiMieSVDXqHDu6Baoa1QYNPFDp+iiTROYhcKIaYQk/ZWZ6i3OVF2kvZpJxcvJU+HZcCGeDRdC7MaBVqtbxHP8vBKZ2XKUSumJY+K19A9+FqUPr6IgZycUVWXgcHjwbB6E2KFvIrLPS+ALqI/zk9QFAF37TISr2BunDqbg2O7/gcPhwL99FEa/vBytn4lxmi5aNOKeNVUY0YD/3xfknmiTjKXRAGk7i5C2s8juWEzVRTIWyXhn81U4m09uBR8Tr2XbTvFo24neiymsQbKOAUBo1GCERg22Ow5pXVTRNoGl9KydLAsLy1NJ9eFUyue69n/BgUocByN64Cws5qisrMTmzZuRkZGBoqIiCIVCtGvXDhMmTEBiYuJTMRMqNDTU7H43NzecO3eugdU8ZThoDDwjIwOpqanIy8tDZWUl8vOfWOP+9ddf+Oabb3Dp0iXI5XIEBQXhtddew+DBln/J7NixAwsWLICrq6thX2hoKNLS0urV0qANuDnf59A2fAyJc8GNu2oj29AeEUJUK7VmbUNXv+MFwLz38N7vOxusK58JckWVXGOz7/YbrzyxID2ZU6KzIB3pj/O55WYtSPUe0pP/z/hn+qShEgzqIcbZy3IU3lZArQbC27ugRxc33LynxEerH0JVZ1LFT0tbE4ulj0eivPRpSMYy90o1jUaD5R8m4+/88+jx3Cg8O2AClAo5Th/fj/nz5yPrVD6Spr5hlMbaK9VsvZYk65g5P3DAsp2sufOt+YHbWv5170tb7klAd1+S1GUvWgf9g5dIJJg8eTLkcjk++OADo2Pl5eUYPnw4li5dCi8vL/z222+YN28eUlNTDe/ONIe/vz9+//132lqc3gMnaRsKMNO29dTFauw6LEW14kmlP3SqEvcf1yBxgAT9Y91x8AQ1JzqSsQBmWvkCwPXCi7h65RziR76AiS+/Y9j/3NAJ+GhuIo4e3GbSgFuCtZOlB1PvSdo4qAfep08fAMCpU6dMjvXrZ/wuzyFDhmDNmjXIycmx2oDbCmPn2dhiGwow07b1+h2VUYOr5+R5Xc+zdUuBU2IBzLTyBQB5le6fkJeP8WpAvkAAsYcXXESu5pKZhbWTJYOz70na0JgHXlFRgaKiIpOtoqKi/u+xwoMHD3Dt2jWEhZl/85Kehw8fonfv3ujduzdmzpyJvLw8SvGd3gPXQ8I2lCQNYUHq46kr/nKZ/T4cJGMxgbYhEXBz98CBnRvQzNcf7UIioFTIcSJrD25eu4IXX/2g/iD/wNrJ2gbT7km60JmFsnHjRqxatcpk/+zZszFnzhybvr+yshJz5sxB//790aNHD4vnxcbGYvfu3QgKCoJUKsXatWsxdepU7NmzBy1btrT6HYxpwEnYhpLE0RakHA6QGO+BGrUWf/5lOgbsrFhMwV0swaz5X2HTN0uwdvl7hv0iV3fMfHc5orpTX6XI2snaBtPuSdrQGEKZNm0aEhMTTfZLJBIzZ9ePVCrFv/71L7Ro0QKff2795cqBgYGGv728vPDee+8hMzMTWVlZmDhxotW0Vhvwu3fvwt/fn4Zs2zlfqML9xzK4CDkI9OWhczCftm0oSahakMpqbKvMU0d5oUOQC9L2l+PeI+rLwh0di0m4iNzg3yYYXWL7ITi0Cypl5Ti87xekfLkAs/7vS4R3jaMUx9HX0h4S5/xi9LlDzBj4+HVA9v6vcPH4ZkTHz2xwTXqYdk/SRUujAZdIJDY31nUpLS3FjBkz0LZtW3zxxRfg8+n3kzkcDiXDQKs5nDVrluFvW39GUIWEbShJHGlBOn6QBEN6inHolAy7s6Q2ayQdi0kU3SzE5wumI7xLHMZNewtRcQPQe2Ai3vvse3h6N8Pmb5dAo6bW4LJ2srbBtHuSNg7yQlGr1VAoFFCpdENJCoUCCoUCGo0GxcXFmDJlCkJDQ7F8+XJKjXdmZiYePHgArVYLqVSKFStWoKSkBH379q03LWU72du3b9cbjCS22IaSxFEWpEkDJUiMlyDrTCXW/2rfEmKSsZhG5p5UqJQKxPQcZLTfxcUVnaN743HxPTwqvkspFmsnSwZn35N00XK4lDc67Nq1C5GRkZgxYwYAIDIyEpGRkcjOzsbPP/+MwsJC7Nu3DzExMYiKikJUVBQ++ugjQ/qPPvoIycnJhs/Hjh1DUlISunbtiqFDhyI/Px8bNmygNPph9d+DNTvZhoCubShJ8gql6B7tg44dJLhw+Yn3sd6C9HwufT/kpIESJA2U4EhOJdZtt+/mJBmLiZSV6Kb1adSmPWONRv3PMWo9cEdcS0eit5P1bdPF2VJMcOY9SRsH+XyPHTsWY8eONXuse/fumD17ttX0S5YsMfq8ePFiLF682CYtNtvJRkdH2/SFdSFpG0oS0hakifEeSBoowbGzlVi7rdQmoyhHxGIqfq3bAwD+PLzbaH9VpRR/nc6Cm1gC31aB5pKawNrJ0oOp9yRdtBwO5a2x4nQ7WZK2oQAzbVsHxblj/CBPFJfW4NJVBXp2cTM6Xi5T49JVagsjSMYCmGnlCwADR07Gyax0/Prj17hz6yqeCeuCSmkFjmXuQHnpI0x+ZT64FKeJsXay9HQx9Z6kDWsn63hI24Yy0ba1faDuvYItvPl4bYKPyfHL1xS4dLW4wWMBzLXybebrj/lfbEbGL2tx5cJpZB8/AKHQBYHtOmD89LcRHUfPxY+1k6Wui8n3JB20aLw9a6o4vQEnbRvKRNvWNVtLsWYrmXFqkrEA5lr5AoBvq0C8NPcTIrFYO1nqMPmepAPdh5ONEac34CwsLCwOoQk04KwfOAsLy1NJyYVjlM/1iezjQCWOo0F74OasJgFgwvAWCA5yxTNtXeHXQogHj5Q2W8ByOMD40QFIGOqPVr4ilJUrcfh4MVJSb5hdrGHNNpQJukjE0scjkUd9PknG2pltOh3wt+2rkPnrNxbTcXl8/GfjBaN9Y2J5Fu1kmVAvFm5QWjxHj4AHzBkjgI8HByevqJF+yrhsPp0uJKZLr62uZXGr5nz0jnJD5xAXtPThQyDg4MHjGpy6WI39x2VQqMz3+X5a2ppofbWbRjy7hCqMGEKZPq6VwS9Y7Grfz565yU98n9N+va3zfR4VgJD2YrO+z41RF1PzSDJWROwgNGvZxmT//dsFOJLxPTpGPUcrHlPLrC7xUTy4i2xLS0rXc93cDJ7zf/xVZfCcnzjEE3GRrhY95y1BsuzpwI6BNxBM9PBmqi6m5pF0LL82ofBrY/q2mu3rFwEAYvslUY7F5DKrjZ8PBz3CuThwRo3hz9K/NUnpIuk5T7rs6dAUZqEw4l8UEz28maqLqXkkHcscSnkVzp/cB0+fVgjt0ptyOiaXmR4OBxjTk4fCO1pcvmWbLwspXSQ950mXPR0ctZSeSTRe5WZgqu8zSV1MzWNDcOH0ASiqZYjpMwZcGsukG0OZ9QznooUnB+mnmLvK0RbPeaeWvYPMrJiE1QY8MzMTqalP3uw8fvx4xMfHIz4+Hvv373e4OLrU5/vs7SkE34yhUWPSxdQ8NgTZWdvB4XAQ28+8D4UlmF5m3mIgvisPh8+rUSZzmgyr2Oo578yy13B4lLfGitUGPCUlBQMGDDB8ViqV2LZtGzZv3owtW7Y4XBxdqPo+NzQkdTE1j46m+O513Cg4i+Dw7vDxbU0rLdPLbHQPPkpkWvyRyxxL27roPee3/VZBy3PemWXf5IdQVCoV/Pz8DJ9jYmLg7e0Nf39/VFdXO1wcXZjq+0xSF1Pz6Giyj2wHAMQ+N452WiaXWZf2XAT7c7DnhBoahq7IsMdz3pllrwWH8tZYsdqA132hZ21P25KSEscosgOm+j6T1MXUPDoStboGOcd3wU3shYhuA2mnZ2qZ8bjAsFgeCoq0kFZr4eMB+HgAXu46nSKh7rNI2ODSDNjrOe/Msm/yPfDIyEj88ssvJvvT0tIQGRnpMFG2klcoBY/HQccOxq9G0vs+5191zgAjSV1MzaMjuXI2C7Lyx4jqNQp8Af3WjKllJuADYlcOwgK5eDtJaNiSh+lmeXQN5uHtJCFiQpzTwJDwnHdm2Td5O9kFCxZg1qxZ2LNnDzp16gQAyM3NhVKpxOrVqxtEIB0OHSvGlPFtMCEhwMi439m+zyR1MTWPjuTJ8An1ud+1YWqZKVXAlsOmU//cRRyM7sFHQZEGOYVq3C9t+F8HpDznnVn2jfnhJFWsNuDNmjVDWloaTpw4gatXrwIA+vXrhx49ehAVwUQPb6bqYmoeScfSU1H6EAUXjiMwuDP8AjvQTg8wt8w0WiD3pmnL6CXW7SuRas0ed7Qukp7zpMueDo4a287IyEBqairy8vJQWVmJ/Hxju4LLly9jyZIluHLlCry9vfHyyy9j6tSpFuNpNBp89dVX2LZtG6qrqxEdHY0lS5YgICDAYho9lJZ79ejRg3ijXRsmengzWRdT8+gI3+czR3+FRqO26eFlbZhaZiQhpYu05zzJsqeDo8a2JRIJJk+eDLlcjg8++MDomEwmQ3JyMiZPnoyNGzfiypUr+Ne//gVfX18MHTrUbLyUlBSkp6fjxx9/RMuWLbF06VLMnDkTu3btApdrPQ+MWErPRA9vgLm6mJpHR/g+D0h4FQMSXrU7DlPLzBxlMmrGV3UhpYu05zzJsqeDo3rgffronAtPnTplcuzgwYPgcrl4/fXXweVy0bVrV4wfPx4//fSTxQY8LS0NycnJaN9e9xrBd999Fz179kROTg5iY2OtamHtZFlYWJ5Krv99lfK5zVr4msy6A3S9bYlEYiaFrgGfOnWq0RDKZ599hhs3bmDt2rWGffv27cOiRYtw+vRpkxhSqRTdunXD1q1bjSaGjBgxAhMnTrQ69AIwpAfOwvK0s2bNGuTm5iI3NxdFRUUICAjA77//bvbc5cuX48yZM7h58yakUimaNWuGsLAwvPzyy+jevbvV7ykrK8OaNWuQmZmJ+/fvw93dHSEhIXjjjTfQrVs3WppDQ03NxADAzc0N586doxXLGdDpgW/cuBGrVq0y2T979mzMmTOHchyZTAYPD2N7AIlEApnM/Gwb/f66/yQ8PDwspqlNgzbgs5abjsGFtuFjSJwLbtxVo1SmgUYD+DfnoUeEENVKLf6zSQpZlfGPhNXveAEw74m89/vOBkvNZ4JcUSXXUPJqJhmrrr/ypKESgz1n4W2FwZ6zRxc33LynNGvP+dPS1sR06dMwNRYAqMrKUJmfD/ewMKgrK3F2dILZ89v/3/toNW4cys+cwePDh8ETidAyMRFCX19whULKQw9Du/EQG8qFi8C67zaVeFRirVixAp7urggLbIUKNxE01TJU/77ZbLyzWQcQ3Ko5+vePhsRNhEflMuw9fQlTp07F559/jjFjxuCHw6bpyh/fwU8rpkCpqEKXXuPQqV9bKKplKL6Tj59/f4CLZtbgvNQfWLHL8o/wVu26oWP3CUb7uFy+2TRvJ3CIlBfwpPztQUPD6mnatGlITEw02W+p920JsViMx48fG+2rqKiAWCy2eD6g64nXRiqVWkxTG6f3wPNv1SD/lunS3KtFNUge7Y64TkJkZlN/yzoT7VFJ2nOS1MWUWDkJY6C4o3Or6/pzGriurmbPc+vQAa3GjUPpH3/iyhtvGPbf374DUdu2Qdi8GTgA6hsTtNe21ZZYmZmZaF6YBQBIWrIGVQrLDd36t01/Nk/u/yxGfrQaa9euxZgxY8ym2/PDu9Bo1Jjx4W6IPcm4/El8WqND9GgisQCyZV8fWhoNuLWhEjqEhYVh37590Gg0hgeQubm5CAsLM3u+h4cHAgICcOnSJXTurOvMSKVS3Lp1Cx07dqz3+xi7BKmkQre81k1E70EEE+1RSdpzktTFlFj6xrs+PLvFAAAeZqQb7VfLZCg5qnsTTBtf6/WFhG2rLbECAwPt+i43kRBe7q5mx2kB4FZhNoqu5qD74GSIPX2hVqugUpKxu1DXKKFSUO9gWIJk2VPBUUvp1Wo1FAoFVCpdfVcoFFAoFNBoNBg8eDDUajW+/fZbKJVKXLhwAVu3bsXzzz9vMd6kSZOwfv16XL9+HVVVVVi2bBnatm2LmJiYerU4vQeuh88DXIQcCHhAq2Y8jOmrey1J7jXHekw7E1vsOZsy3H9WYWrkpr/INHI5ACCwBQc3H1rug+ttW7dk2V+vSMYyR6msClqNFsUVMuw4fg7X7j9CUpL5xUzXLun+gUm8/bB19Uxcyz0KrUYNb9+26DXidUR0Nz8kVR/XLh5E4bk90GrUELn7ILjLMMQOeRMurvRtYB1dXnVx1CyUXbt2Yf78+YbP+oePmzZtQvfu3ZGSkoLFixdjzZo18Pb2xqxZszBs2DDD+cnJyfD398eSJUsMn6VSKSZPnozq6mrExMTg22+/rXcKIcCgBrxXpBAT4p8sGHhUrsaGjEr8fcd0fOxpwFZ7zqZM1bVrAADPbt1QevSo0TFJdLTumLvlm7aubatX/UOMDRLLHFVyJfq/u8LwWSTgI6l3FBYuXGj2/Mf3dVMI96d+CO8WQRg5bSnUahVOZ/6A9B/eg0Zdg8ie9Fay+gZGon3kEEiaBUGlkOFW3lHk/pmKe9eyMWbWFghc3CnHcnR5mcNRDfjYsWMxdqxlS+Pw8HD8/PPPFo+npKQYfeZyuZg3bx7mzZtHWwtjGvDzhSrcfyyDi5CDQF8eOgfz4e7aeD0K6kNvz5m2v5yWPWdTpvTPP1H19zW0Gj8OykfFKPn9MLgiEfxfeAFuwcEAdP4iliBp2+poC1gXIR/fzZ0MtUaLeyVl2Hv6EqoUSsjlcri5uZmcr/xniEPo4o7Jb28Cj6/7tdKhy0B8++FAHNn5JTrHJYJDoVenJ3GOsQ9Sh5gx8PHrgOz9X+Hi8c2Ijp9JOZYzLHMbs8sgVaw24J988gk4VoxeLPUGbKFMpkWZTNeQXbiqwrkCLt570QNCPgcHT1N/iNkYsMees0mjVuPy3LkIWfwx2s6di7Zz5wIAKgsKcHPVKrR76y0oLPw619u2rt9XY7dtK8lYluBxuYjr2N7wObFXFJJXbMa0adOwY8cOAMbPTfgC3ZBjeOwIQ+MNACJ3T4REDsClkzvx+MF1NPcLtktXl34zkPPbatzKy6LcgDdEeZlDo2XsIz5iWG3AIyIiDH+vXLmS1nxIe7n7SIOih2r07eryVDXg9tpzNnWUDx4gd+ZrELZsCZG/P1Tl5ai+dg2txumW2heXm7YQ5mxbAUDiZmzbWqUA5PXMgiMZiw48LhfDn43Av7fsw5kzZwAYW1tIvFsCANwlLUzSij11++RV5SbHaOvgCeAu8YW8ktpKTWeVF8D2wI3mRW7cuNHsPElHIuBz4PYUDaOQsOdk0aF88ADKB0+c7Lx69QQAXL1j+hP9iW0rB2GBpvOLuwbz0DWYh33ZNfX+xCcZiy7yfxYLlJWVASLjY35tI3HuaBqkZfdN0klLdfvcPZrZraFGpUBl+QP4tulC6XxnlleTb8BrY20oxR4kbhxUVJn2mkIC+fBvzkXh7adjfJiUPSeLKd59+8LnH3+KMjOz3UjatjraAraishquLkII+MZWqNUKJXb+8Re4XC4iIyNxv8A4XYcuA5Ep+jdyT+1Gz2GvQSjSPWCUlT9EwflD8GnZFt6+QZR1yCtLIXL3NtmffeB/0GhqEBTen1IcZ1rmarVsA+5wJg5yhac7FwW3alBSoQGfD7RpyUdMmAByJbAjS04rHhPtUUnac5LUxZRYLYYPg8s/r+7je3mBKxCg9YyXAQCKe/dQvHef4dzgDxcCHA4q8wugUSgg6doFLYYOhTQ3Fx7/eNbXhaRtq62xdu7ciVvHjwHQTQ9U1aixbq/us18zT4zsrpuKdqbwFj79aS/io8LQpoU33ERC3HlUhozTF/GgVIrZs2frbEbrNOAid08MSHof+1M/wqYvJiKyZxLUNSqcO7oF6hoVBk38kFL+9Jw99B0e3PoL/sHd4eHlD5WyErfyjuLu36fg26YLInq9SCkOactcOmiaeg88KirK0POWy+WI/meqllarBYfDwdmzZ+0WkJOnwrPhQjwbLoTYjQOtVreI5/h5JTKz5SiV0ru4TLRHJW3PycQ82hPLNyEBnnUWLbR57TUAQHlOjlEDLsvNRcvERDQbMAAcPh/yoiLcWrMG937agrg/jlPW2tBs377dxMxo9R7d3O2YkDaGBjwkoAX6dQ5BTsFN7Dt9CXKlCp5iV3QK8scHzw/HECvPobr2mQhXsTdOHUzBsd3/A4fDgX/7KIx+eTlaP1P/opDa+Ac/i9KHV1GQsxOKqjJwODx4Ng9C7NA3EdnnJfAFLjRLoOFp8g8xG8Kw5my+CmfzyU3sZ6I9Kml7Tibm0Z5Yua9Sn472YMeveLDjV5u+py622rbaEmvz5s0WvU9qE9jCB4umjLRZR2jUYIRGDbY5vZ62neLRtlO83XEsQbLsLcGOgbOwsLA0UprCGDjrB87CwvJUkp1PffgvNtTLgUocR4P2wOvarAJAq+Z89I5yQ+cQF7T04UMg4ODB4xqculiN/cdlUKhM/7/orVZ7jzpicozDAcaPDkDCUH+08hWhrFyJw8eLkZJ6A3KF6TSl43v6PfWx9PHMWcBOGN4CwUGueKatK/xaCPHgkZKSBaw5Xbbmk8pPaQEPmDNGAB8P6xakJHUBpnXWnvpqzgIWAKory3Bi3xoUnM+EtPQ+hCJ3tPAPQZ9RbyAwxNjD+6V/Jn84+lraWsdI3+P20BR64E4fQnmum5vBK/uPv6oMXtkTh3giLtLVrFe2NeYmB2P86NY48mcx0n69jaBAd4wbFYCQ9mK8+eEFWtP3mkKs6eNaGTy8xa72P/Qhqa028VE8uIvqP8/RwuBM1AAAIABJREFUukjX17oe3t6+Tzy8pWX03thO8lqSvI6ky4wqTcEizukNOEmv7HZt3JA0MgBZfxZj4X8uG/bfe1CNt14NwcC+vpTfgt0UYgFk/cBJa9Njr4c0SV2kvd1JeniTupakryPpMqNKU5iF4vQckvTKHtjXF1wuB7/sMvaX3nPgHqrlagx+jvoN0hRiAWT9wElrA8h4SJPURbK+kvbwJnUtSV9H0n74VNFqOZS3xorTe+CWsMUrOyzEA2q1FlcKjE3vlSotCq/JEBZC3cO4KcQijSO0kfCQbogys6W+OsrD214aqo452g+/KUwjdHoP3By2emU39xGivEIFVY3pf/tHJQp4ewrB51O7qE0hFmlIa6vrIc0UXXWxtb7W9vCWV5Zh5LSlGD71M/D4AqT/8B4u/LndZk320BB1rCH88DVa6ltjhZENuN4re9tvFbS8skUuPKhqzP83Vyo1/5xDLctNIRZpSGsj5SHt6DKztb7W9fDu1H00Insm4cV5qXBxk+DIzi+h1TT8o7iGqGO2lhkdHPVKNSZBeSl9bUgupa+LPV7ZcoUa3iLzb7MWCrn/nEPthmgKsUhDUhtJD2lHlpk99bWhPLzp4ug61lB++I15bJsqTl9KXxt7vbIflSjRNtAdAj7H5Odfcx8XlJYrUWPmZ2FTjUUaUtpIe0g7qszsra8N5eFNF0fWsYb0w1c7qAEfMWIE7t69a/is0Wggl8uxatUqDBo0yOT8AQMGoLi4GHz+k+Z2xYoV6N+fmqOjNRgzhELCKzuvUAoej4OOHSRG+4UCDkLai5F/lfpAalOIRRpS2p54SHPxdpLQsCUP081W6BrMw9tJQsSEUKu+jigzEvXVr63OwMrRHt50cVQda2g/fEfNQsnIyMC5c+cM27x58+Dl5YW+fftaTLN48WKjNCQab4AhDTgpr+xDx4qh0WgxISHAaP+oIX5wFfFwMIv6woimEIs0pLTpPaTrbrtP6MZKC4o02HJYhbzb1H7Gky4zUvW1Q5eBEIrckXtqN5TyJ/OgbfXwJoUj6pgz/PC1WupbRUUFioqKTLaKiop6v2fLli0YN24cXFwa3qHR6dMISXplX7tZiR0ZdzFuVAD+PT8cJ3JK0La1G8aNCsC5i2W0Fh80hVgAWT9wUtpIe0iTLDOS9ZW0hzepa0m6jpH2w6cKnYeTGzduxKpVq0z2z5492+qrJE+cOIEbN25g0qRJVuMvW7YMS5cuha+vLxISEjB9+nQIBPbPf3d6A07aK/vrlKu4/1CO0UP80CO2GcorVNiWfhfrU6/T/q/fFGKR9AMnrY0kpHSRrq8kPbxJXkuS15F0mVGFzsPvadOmmX1lpEQiMXP2E7Zs2YI+ffogMDDQ4jlLly5FeHg4RCIRLly4gHfffRdlZWV49913qQu0gNMbcNJe2RoNkLazCGk7TU112FimkPQDB8hqq4s9HtKkdJGurwA5D2+S15LkdXREmVFBo6HeA5dIJPU21nV58OABDh06hNWrV1s979lnnzX8HR0djblz52LZsmVEGnDWTpaFheWpJP0s9fnlI6Pp92VXrlyJnTt34rfffgOXS/1x4p49e/D555/j+HH73yDl9B44C4s5Vq5caXZMUg+fz0dubm4DKiJPU8ijM3Fk17Smpga//PILpkyZYrXxvnHjBoqLixEZGQmBQIBLly7h66+/xogRI4joaNAG3JJX8xuvPLGuPJlTorOuHOmP87nlZq0r9V7N5jyR937f2WCp+UyQK6rkGqueyHu/72xRm626SMaq6688aajEYM1ZeFthsObs0cUNN+8pLVpz/rS0NZHy0qcheS13Zpt6e/P94zFxpqkn9P3bBTiS8T1Cuz5nkm5MLI94HSN5LevqtTWPAJm6r09Dso6RKC/gSZnZgyMX8hw6dAhlZWUYN26c0f67d+9ixIgRWLduHbp164aKigp88sknuH37NjgcDlq2bImkpCTMmDGDiA6n98CZao/KVAtY0tacTLWT9WsTCr82oSb7t69fBACI7ZfkFF1MzSNA7loy1eKZLo70OBkyZAiGDBlist/f399oAWRkZCR2797tMB1OnwfOVHtUplrAkrbmZLqdbG2U8iqcP7kPnj6tENqlt1N0MTWPALlryVSLZ7rQmQfeWHF6A85Ue9TGZgHraGtOKjg6nxdOH4CiWoaYPmPA5fKcooupeWwInG3xTBe1lkN5a6xYHEKxZGQFAEKhEG3atMFbb72FHj162CWgPuvKyHBP8PmcBvf3IKnL0XlsCGtOKjg6n9lZ28HhcBDbb6zTdDE1j47GURbPjry/G3PPmioWG3BrRlZqtRqFhYV45513kJ6ebpcAqtaVshrTB12OhKQuR+dRb82Ztr/cYdacVHBkPovvXseNgrN4plMcfHzpvfC2sVxLe/LoaGytY868v5tCA27TEAqPx0NYWBhefPFFuwXIFWoI+OZlONMelaQuR+axoaw5qeDIfGYf0b3cIPa5cfWc6VhdTM2jI7HX4tlZ97dGy6G8NVbsGgOvb/0/FR6VKOEpEUBg5g0fzrRHJanLUXlsSGtOKjgqn2p1DXKO74Kb2AsR3QY6VRdT8+goSFg8O+v+Zh9iNgBMtUdlugVsQ1tzUsFR1/LK2SzIyh8jqtco8AXmXzTQULqYmkdHwDSLZ7qoNdS3xorTG3Cm2qMy2QLWGdacVHDUtXwytEBvXrQjdDE1j6RhosUzXdi30jcATLVHZaoFLGlrTibaydamovQhCi4cR2BwZ/gFdqCdnrQupuYRIHctmWrxTBemdGwcidMbcIC59qhMtIAlbc3JdDvZM0d/hUajtvvBHhOvpR5SeSR1LZls8UyHxvy2eaowogFnqj0qEy1gSVtzMt1OdkDCqxiQ8KrdcZh4LfWQyiOpa8lki2c6sD1wFhYWlkZKU2jAWT9wFhaWp5J1mdTPfYU5Mzdp0aA9cHO2ob9tX4XMX7+xmIbL4+M/Gy8Y7dNbapqzruRwgPGjA5Aw1B+tfEUoK1fi8PFipKTeMLtgwJo17YThLRAc5Ipn2rrCr4UQDx4pKVnT1rXnbNWcj95Rbugc4oKWPnwIBBw8eFyDUxersf+4DAqV6f/Qn5a2JqZLr41EeQG6MqubR3vyWfjpajQf1Atu7duAJ3JB5bVbuL9tP65/vRHqqmqj8907tEPYZ++gWd9YcIQCVJy7jILFK/E46yRGqPLNlpctZUb6Wq5ZswYX9vyMKw/LcKe8En4SN2QkDzf73Yv2Z2PP5Ztmj/3vf//D0KFDsWKX6feseS/MbBq+0A0zPj1r9tjbCRyzsWyJ93YCh2gdsxdNI54eSBWnD6FExA5Cs5ZtTPbrPZE7Rj1HK97c5Cfew2m/3tZ5D48KQEh7sUXvYUtMH9fK4K8sdrV9xuVz3dwM/sp//FVl8FeeOMQTcZGuFv2VHa0LIFtetuaz3RvT8SD9d9zdsgcaVQ2a9euO0E/egt/4Yfij1wRo5LoZD27tA9Hz6BZoa9T4e3kKaipkCJwxHs/uTcHpka9Y1ebsa7lixQp4ioQI8/WCVE7ttXCfDI012RcZGWk1Tat23dCx+wSjfVyu7bc5iXgk6xgdmsLYgtMbcKb6PgPM9FcmqYt0edmaz0Nt+6Km4smCjltr01B59SZCFryGwJfH4eY3qQCA0E/nQeAlwfHuY1FxPg8AULR5J/qeT0fE14usanP2tczMzITP3u8AAOM3HkQVhf/YI8KDTPa5+/tbTSPxaY0O0aPrjU0Ve+M50w+8KTTgTl/IYw4m+D4DzPRXJqmLdHnZms/ajbeee1v3AgA8OunmRfPcXNFy1AA8PnLa0HgDgLqyCre/3wZxaDur2px9La29tdwSWq0WMoUKGpotkbpGCZWCeofAkfGc6Qeu0VLfGitO74GbQ++J3Gvwi07zfW4InO3h3VDlZUs+RQG6+cuKB48AAB6RoeCJXFB68i+Tc8tOme5raBxxLfuu3oVKZQ0EPC6iA5rj9V6dEFdPmmsXD6Lw3B5oNWqI3H0Q3GUYYoe8CRdX266lvfGceU/Sm59BfTXmypUr8c0330AkEhn29e/fHytWrDB7/u3bt/Hxxx/j7NmzcHV1xfjx4/Hmm29atOumg9UG/N69e/Dz8zN77PDhw+jfv7/dAszBBN9nR8MED++GKC+b8snlIuSD16BRqXA3TWdXLPLT9dTkd02XXpvb15CQvpbN3EV4IToEHVt6w1XAQ0FxOX46W4gZP2dh3eA/0bNnT7PpfAMj0T5yCCTNgqBSyHAr7yhy/0zFvWvZGDNrCwQu7rR0kIjnzHtS7UAH6m7dumHz5s0UNKgxc+ZMREdH4+uvv8aDBw+QnJwMiURC5L2YVhvwl156CSkpKWjd2tibeNu2bfjuu+8c0oAzxffZ0TDBw7shysuWfHZasQDePaKR98F/UVmgW5zCc3MFAGgUpg8A1XLq1gGOgPS1nNuns9Hn/s8EYFhYICZtzsTHH3+MgwcPmk2XOOcXo88dYsbAx68Dsvd/hYvHNyM6fiYtHSTiNRY/8IqKClRUVJjsl0gkkEgkZlJQ48yZM7h58ya2bNkCd3d3tG/fHsnJyVi/fj2RBtzqGPj8+fMxY8YM3Lhxw7BvzZo12LhxI3788Ue7v9wcTPF9diRM8fB2dHnZks8OH7+BtrOm4Oa6NPz9xVrDfv10Qq6LqVMfT+Ris0Z7aahr2cbbA4NCW+PmzZu4fp36issu/WaAyxPgVl4WER104znXD5z6tnHjRsTHx5tsGzduNBv70qVLiIuLQ//+/TFv3jzcvn3b7Hl5eXkICgoy+icQERGBoqIiyGT2OzFa7YH369cPQqEQr7zyClavXo2tW7fi4sWL+PHHH+Hp6Wn3l9eFhO9z2/9v7+zjoqryP/6584DAwCAPCzo81CtEwRJFkMBWbOuXmbiWtZLuxropplvY/l6tlLvtbkq5PlC6tZWYmo+Z/Yj1EdwIy10fMp82SAShVgSiFBEYhmdm7u+PccbGe2eYO3PuzB0579frvl5y594v55x7+Xrn3HPeJ1IFpYLhfGVzp1v8x0jJ4S1mezlSz5g/ZyPm5WdRv7UQ55+1HFXS/b1xpIK3JoxzHt8+V+Dqa6lRG7ssWlpaANxp1zlyuRIqdSi6O8hMjRcaz51/k0KewOfOnYuZM2dy9vM9fT/88MN4/PHHodFocPXqVbzxxht4+umnsW/fPqhUlt1KOp0O/v6W/fymmDqdDn5+lu4aoQw4CiU1NRUrV65EZmYm6uvrsW3bNlGSNyAt77MYSM3hLVZ7OVLPmD9nY+RfFqN++z9Q/szLnM/bv66GvrsHgSnjOJ8NvZe7T2zccS3rWozXIyQkxO5z+vt60NF2BT5+9p9DMp47/yZZA2v3plarERERwdn4EvjIkSMRHh4OhmEQFhaGFStWoKmpiXcZSj8/P86TtqmrxtnkDQyQwBMSEjB+/HgsWLAAHR0dOHnyJFJTU837SSMl7zNppOjwFqO9HKnniJefw8i/LEbDzr0oz/oj76OTvqMTVw9+juDJyfCPvzlvQK7yReS8X0BXTVbKZQsxr2VXXz96ePqDq662oLSmAdHR0YiK4k58s/ZEfPqTN2Ew9OOO0cLeV5GK586/SVct6MAwDBiG4R31Ehsbi8uXL6O9/Wb3WkVFBSIiIogkcJtdKLYWNiaN1LzPgDT9yiTLRbq9HK3nqGXPo/Pyd7h2+ATC5/zc4rOeK9dw7fAJAEDVn95A8AMpuLf4fVx6cyv62zsQOX8WvMPDcHrGQtxbvNlq2dx9Lffu3YvLJysBAC1dPejTG7Dpxs/D1L6YfmPSTl2LDov3HMP90RpEBfrBW6lATVMr9p2vhZxhkJuby1uuc4fzcaXuK2ii74X/UA36ejtQV/VvNH77JUKjxuKe+4StX0sqnjt94AaRBngXFxcjJSUFQUFBaG5uxuuvv46goCAkJCRwjk1KSkJUVBTy8vKwdOlSXLlyBZs2bcKcOXOIlEUy48Cl6H2Wql9Zqs5zZ+rpe0c4xm1Zw9nf/K8vzQm889s6fDF5DkatWILoF5+BzEuJtv9cwKn0LDR/9oXNsrn7WhYWFuLUqQqLfe+eMP6cGBFiTuDBKm8kR4XiTH0TDlXVoadfjxCVN6aMisTTyaNwT1ISb7k00cloufoNqs/uRU9nKxhGjoCQOzBh6v8iftLTUCiFveglGc9dPnCxYu/fvx+5ubno6uqCWq3GhAkTsGXLFvj5+aGxsRHp6enYuHEjkpKSIJfLkZ+fj2XLlmHixInw8fFBRkYGkREogItthHwyK0ewJbMSii2ZlVCsCZAcwZbMyhGsyawcwZrMyhF2rYpAkZKrUnAEWzIroYhxLTs2cPv2haJauAIArAqohGJLZuVILJL3mLOs2G1/vnl5tv0TBqUE1clSKJTbklc/tH9M/p/nSKYzQhCeWWoKhUIZANb9Uz5Ex6UJ3NrXq98tuKmbPHn2ulE3OV2Dsoo2Xt2k6esVXzwai1/Paa3bY/ZUtVmPWlPfY9ajpo71xeXve3n1qLtWRfB2VRS/P8asbB1xhw86uw12ecr/tHVgverUJDkmjJJhiJLByUo9Dn7J/Xr82m+8iN9jz71u2Tc+KkqBh1OGoLZRjxadAQYDoAmRI/UeL3T1sli5vR26Tstg7ywZCgBE6vnab4x98GLeF47cEwD5+8JZ9Prbv3PB7U/gJHWTNJbrFLB8kFK23srwIAapo2X45Iwe05KF37Ik2+xiXT8u1nGz1zcN/ciaoULK3V4oPe3Y1H5n6kmyjlLVHwtlMPQOu10nS1I3SWO5TgHLByll649hGOCxiXLUfMfiQp1j34ldoTS9rjWWzdfbMcOcs/UkWUep6o+FQnWyLoCkbpLGcq8CVgwmjpbhJwEMPjzieBIQo80UcmCIFwOlHBgWLMdjaUa1aMV/HSuns/V0xX0hlXvCXlhPzsx24vYn8IF0k4EBXlAo7HuqobGExbKGFFS3ABDoBzw4To7Py/RodWLGtRhtdl+8F9Y8F4AViwKweJYffLwZbC3qwLffCR8qS6KeYt8XUrknhMCy9m+eisMJfOvWrUQKYK9uksYiH8saJj3qx59q3aa6BYAZqQpc17E4XuHcE58YbVZW04e3/k+HDXs7UHyiG13dLFQ+jiVIEvUU+76Qyj0hBIOBtXvzVNyewEnqJmks9yhgxWDsXTJEaxgc+ELvdB+lGG3WqmNxsa4f5d/0oehEN7Yf6sRjaT6YkixsxiOpeop5X0jlnhCKQc/avXkqDidwUm94r13vRYBaCSXP1zuhukkay/UKWDGQy4BHJshR3cCivYtFkD8Q5A8MVRnr7O1l/NnbTmGlmG1movGaAQ1X9UgbZ38CJ1lPseoolXvCEQwsa/fmqTicwEms5waQ1U3SWK5VwIqFUgH4+TCIjZThhSe8zFvWI8bRD+Oi5XjhCS8kxth3+7pKaapUMPAV0I1Csp5i1FFK94QjsCxr9+ap2ByFkpCQwJuoWZZFTw+ZZawOH21C5qwoZDwajvILbeb9jugmaSzXKWDFpLcP+PBz7mgMlTeDGakKVDcYcLZGjx9a7CsoyTZT+zLQdnJ/b0ykApoQGWrq7e8fJllP0veF1O4JR/Dkvm17cbtOlqRuksZynQKWD1LKVgMLVFzm/vEN9TPuu97O8n5uDZJt9uRDPghQyVBd14/rWgMUCiAqTIHEWCW6e4F/HOm2OxbJepKso1T1x0LxxP90hOL2ceAAWd0kjSUsFknVLUnNLWlItdnZqj4kj/ZC8mgv+PkyYFnjJJ5jZb0oPd2Nlnb3ZQ1SdZSy/lgIemdXavAAJJHADQZg994G7N7rvLqTxhLGhoIWbCgg07+5dI24q+K06uzzifBBqs3OXezDuYvizix0tJ6k6kjyngDEvy+sMRgm8kgigVMoFAppBkMCpz5wCoVyW/LbPPu7ZtbnDBWxJOLh0idwa6ulZEz7CaLv8MGIO30w/CdeuHKt16Zu0qSa5FNqMgwwa0Y4Hp2qwbBQb7S29eLzY03Y9EEt70QGWyvyOFoukrFI1NFUT5Kx+NS0w0IU+GmCL8bEDEFYkAJKJYMrzf348usu/POYDj193GeFXasiOMpWAAgNlGFaqjciw+QI8JNBLjP2NVdc6kPp6R5oO7ix3lkylGgsgNy13LBhA8oPfITKq634rq0Dw9W+KMqaxnv8K/88jQMXLvN+9uabb2Lq1Kmc9nek7QFj+/OtlPVp4dso3fOu1TrJ5Aqs3FZuse+xCXIi7QWQ0cmK9QSel5eHI0eO4Pvvv4evry/S0tKQk5ODwMBAq+c88MADaGpqgkJxM+WuXbsWP/uZsMWmb0USXSi/+cUwsy/Yz8e5aeDPZ910Iu/eU290Iv88HDF3+Vl1IruiXFKtI8lY9yf5mj3Sx7/qNHukn3w4ACnxPlY90nwE+sug9mNQVtNn4d3+afwQJMZ68Xq3XRELcOxarl27FgHeXogNHYr2bvv6t1+dOoGzLz4+nvdYkm0PAPdMeAjBYVGc/T/UV+NfRe8jLuF+u2ORvPeFIFbnglwuR15eHmJiYqDVapGTk4M//OEPyM/Pt3ne8uXL8fjjjxMtiyQSOClfMGlXNkmPsRTrKGW3OEnvNmmHtyPXsrS0FEHFxj/wWdtK0GlHNk2/sdDxj1FpNLzHknZ4D48aheFR3LVKCze/AgCYMPkJu2O5ywcu1iiUF154wfzv4OBgZGZm4ve//70ov2sg3G4jBMj5gkl7n0l6jKVYRym7xa3hrHebRCxHrmVkZKTgc1iWha6nz66p3q5o+97uTpSdPISAoGEYNfandp/nLh84a2Dt3rRaLRoaGjibVqsd8Pd88cUXiI2NHfC4vLw8JCcnY/r06di4cSP6+pxvF0k8gZPCVa5sdzLY3OIkvdukHd5ik/bOPnT09kMpl2F8eAieve9upAiMQdLhXX7qE/R06XDflKcgk0l/FXchfeDbtm3D22+/zdmfnZ2NxYsXWz2vuLgYBQUF2Llzp834q1atwujRo+Ht7Y3y8nLk5OSgtbUVOTk5dpeRj9sqgQ/kRI4fHQCFgnFaXOROSNbRFe3lrEf6vngvZDx4cybgtTa9w95tkrHEJFjljV+Nj0FcWCB8lHJUN7Vh17kazP/oCDZOOYGJEyfaFYe0w/v0kUIwDIMJk8n244qFEEnV3LlzMXPmTM5+tVrNc7SRoqIiLFu2DOvXr8fdd99tM35ycrL53+PHj8fzzz+PvLw8cRP4okWLbJ48UKe9q7HXiazrl9YfrBBI1tEV7WXySO/+Z5tDHumymj780KzDEC8GkaFyjIlWOOzdJhlLTJ6fZDkC42cjwvFIbCRm7yjFsmXLUFJSYlccZ9v+xzQ1XkJt9TmMuDsFQaERTsVyFUKewNVqtc1kfSsFBQXIy8tDfn4+EhMTBZdNJiPTe20zgX/11VcYPnw40tPTMXbsWMlbu7p79Ai04t4k4cqWAiTrKHZ7kfBIt+pYtOqMyaf8mz78p1qGF5/yh5eCQckpYUI1krFcTVSgPx4aFYEDFZdx6dIlALb7tEk7vE//qxAAMOH+Xzgdy1WIla+2b9+Od955B5s3b8aYMQMPd6ytrUVTUxPi4+OhVCpx/vx5vPXWW0hPT3e6LDYT+PHjx3H8+HEUFRXh4MGDmDx5MqZPn46YmBinf7EYXLveizsjVVAqGE63ACnvs7shWUcx20ssj/SPvdvOJl2SsVyBRq0CALS0tACw/oKZdNvr9f04e2wffP2G4p6k/3E6nqvQW/l26SwrVqyAQqHAr3/9a4v9RUVF0Gg0aGxsRHp6OjZu3IikpCRotVq8+uqrqK+vB8MwCAsLwxNPPIH58+c7XRabCVwulyMtLQ1paWno7e3FwYMHkZmZiezsbDz11FNO/3LSVNW0497xQYgbqbZQapqcyGUVbTbO9gxI1lGs9hLbIy3Uu+2qWGJT12J0eoeEhFg9Roy2rzx3BLq2Ztz3cCYUSjtX0ZAAYj2BX7xoexKSRqOxMLnGx8dj//79opRlwI6Y3t5elJSUYMmSJfjggw+QmZmJhx56SJTCOMvho00wGFhkPBpusd8ZV7bUIFlHMdqLlEda7cufVE3e7dpG+/t0ScYSm66+fvTwvHOoutqC0poGREdHIyqKO8EGEM/hfbP7xP6x31KANRjs3jwVm0/gL774ImpqapCWlobs7GyMHDlSlEKQ8gWTdmWT9BhLsY5SdouT9G6TjAU4di337t2LyycrAQAtXT3o0xuw6cbPw9S+mH5j0k5diw6L9xzD/dEaRAX6wVupQE1TK/adr4WcYZCbm8tbJtIObxPalquoLj+GyOgxGB7p2N+/u3zgg35Bh/3798PHxwe1tbXYsWOHeT/LsmAYBufOnSNSCJK+YJKubJLlkmodpeoWJ+ndJu3wduRaFhYW4tSpCot9754w/pwYEWJO4MEqbyRHheJMfRMOVdWhp1+PEJU3poyKxNPJo3BPUhJvmUg7vE2c+fceGAx6p15eussHLvVBFySwmcCrqqpcUgiSvmCSrmyS5ZJqHaXqFifp3Sbt8HbkWu7YsQMdG14e8LgQlTdeeyR5wONuhbTD28QDjy7EA48udCqGu3zgBpFeYkqJ22oiD4VCoZgwsLd/Aqc+cIrb2bBhAyoqKlBRUYGGhgaEh4fjs88+s3p8WVkZ1q1bh7KyMjAMg4SEBCxZsgRxcXEuLDVF6szMrrH72D1vS3No9EC49Amcz0cNAL9bcFNpevLsdaPSdLoGZRVtvEpTk8ObL54UYvE5kYvfH2NWao64wwed3QaHfeBCy2UqG4lymc4hGWvt2rXw9/fHiOhotLa2oq+vD//99lve4yurqvDSSy8hODgYT/3qVwCA/QcOYM6cOfjoo48wf8kPvOdJ4Vre6vCePVVtVsDW1PeYFbCpY31x+fteXgXsrlURxMplOkeqsZxlMKyM8qdEAAAFrUlEQVTI4/YuFKnqUaWqppVquZyJ9f7mzRg+fDgAYNFvf4uuri6rx+bn50OhUCBvzRrzeOhJkybhmYULsXr1agBzOedItc1IK2ClcC3FjiWEwdC54HadrFT1qFJV00q1XM7EMiXvgWhsbER1dTUmTZpkMZklJCQEkyZNwokTJ2Do504+kmqbkVbASuFaih1LCAaDwe7NU3F7ApeqHlWqalqplssVVFdXAwDieNzLsbGxYFkWfd3cpcg8rc1IKmAHMwa93u7NU3F7Ah9IaRoY4AWFwr6pzlKNRRKplssVNF+/DsC4CsqtmPYZ+rljij2pzUgrYAczQhZ08FTcnsDtVZp6ciySSLVcrqCn2zhjUqnkdi143djHstz1Jj2pzUwK2I8/1TqtgB3sDIYEbvMlJt8KFSYYhsFzzz3ndAGkqkeVqppWquVyBUO8jSvo8C1F1XtjH8Nw28ZT2oy0AnawMxjGgdt87PD19eVsDMOgsLAQmzZtIlKAa9d7EaBWQsnzFdYRPaoUY5FEquVyBcFBxinizc3NnM9M+2SKoZzPPKHNxNLvDmYGwxO4zQQ+b94885aRkYHu7m4UFhZi2rRpKC0tJVKAqpp2yOUM4kZaroZhUppe/Ebn8bFIItVyuQKTTK2SR/FQVVUFhmGg9Oau5C71NhNbvztYGQw2wgE7/lpbW7Fu3TrMmDEDer0ee/bsQU5ODu+LJEeQqh5VqmpaqZbLFWg0GsTExODo0aMWT+HNzc04evQoUlJSIFMEcM6TcpuJpYClDI5RKDb7wFevXo1PP/0UGRkZOHDgAFQqFfECSFWPKlU1rVTL5Uysw4cP4+pVY7nb2trQ39+PDz/8EAAQGhqKBx980HzsooUL8dLSpViSk4MZM2YAMFozWZbF0qVLkZXDTcZSbTPSClgpXEuxYwlh0Otkt2zZAi8vL6xfv95iAWPSOlmp6lGlqqaVarkcjfVJSQm+/tpyCvb2G/riMWPGWCTw0aNHY83q1di2fTu2b98OhmEQFxeHl//4R8TGxgLgf5qWYpuRVsBK4VqKHUsIYnaNGAwG/O1vf8PHH3+Mrq4ujB8/Hrm5uQgPD+c9/sKFC8jNzUVlZSUCAwMxb948zpJsjiAJnaxU9ahSVdNKtVyOxlqzerWg4+Pi4rBq5UpB50ixzUgrYKVwLcWOJQQxX05u2rQJBw8exM6dOxEWFoZVq1Zh0aJF2LdvH2fFeZ1Oh6ysLPzyl7/Etm3bUFlZiWeeeQahoaGYOnWqU+WQxuBXCoVCIQzLGuzehLJ7925kZWXhrrvugkqlQk5ODi5duoSzZ89yji0pKYFMJsOzzz6LIUOGYNy4cZg1axZ27drldB2pTpZCodyWWLOf8lH8QQK0Wi1nv1qthlptOXqpvb0dSUlJKCgoQHx8vHl/eno6nnzySU7XyF//+lfU1tbivffeM+87dOgQXnnlFZw6dcruMvLhdhshhUKhiIFJCWwPf//733knLmZnZ2Px4sUW+3Q647DTWxO7v7+/+bNbj/f3t/TtqNVq3mOFQhM4hUIZ9MydOxczZ87k7L81SQOAn5/xhWx7u+Vs2fb2dvNntx5/6+QzrVbLe6xQaAKnUCiDHr6uEmv4+/sjPDwc58+fx5gxxoUn2tvbUVdXx7sqVGxsLA4dOgSDwWB+wVlRUXFj1JRz0JeYFAqFIpDZs2dj8+bNuHTpEjo7O5GXl4c777wTiYmJnGOnTJkCvV6P9evXo7e3F+Xl5SgoKMCcOXOcLgd9iUmhUCgCMRgMWLdunXkceGJiIpYvX46IiAicOXMGCxYsQFFRETQaDQDjOPDly5ebx4HPnz+fyDhwmsApFArFQ6FdKBQKheKh0AROoVAoHgpN4BQKheKh0AROoVAoHgpN4BQKheKh0AROoVAoHgpN4BQKheKh0AROoVAoHgpN4BQKheKh/D/MbmGRj8BgJAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MtUi5LBjIJTs"
      },
      "source": [
        "## **Model 3** : LSTM (variable-length inputs + random initialization of word-embeddings)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lw5bAq1FquYa"
      },
      "source": [
        "# this model accepts variable-length inputs (number of words in the sentence)\n",
        "class LSTM_variable_input(torch.nn.Module) :\n",
        "  def __init__(self, vocab_size, embedding_dim, hidden_dim) :\n",
        "      super().__init__()\n",
        "      self.hidden_dim = hidden_dim\n",
        "      self.dropout = nn.Dropout(0.3)\n",
        "      self.embeddings = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)\n",
        "      self.lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first=True)\n",
        "      self.linear = nn.Linear(hidden_dim, len(tag_to_ix))\n",
        "      \n",
        "  def forward(self, x, s):\n",
        "      x = self.embeddings(x)\n",
        "      x = self.dropout(x)\n",
        "      x_pack = pack_padded_sequence(x, s, batch_first=True, enforce_sorted=False)\n",
        "      out_pack, (ht, ct) = self.lstm(x_pack)\n",
        "      return self.linear(ht[-1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a_JxrXK7JU7F",
        "outputId": "a424c28a-01c9-45bd-a642-80ebf2931bcf"
      },
      "source": [
        "model_variable = LSTM_variable_input(vocab_size, embedding_dim = 50,\n",
        "                              hidden_dim = 50)\n",
        "train_model(model_variable, epochs=200, lr=0.01) #Faster convergence"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train loss 2.600, val loss 2.562, val accuracy 0.168\n",
            "train loss 1.758, val loss 2.424, val accuracy 0.267\n",
            "train loss 0.716, val loss 2.928, val accuracy 0.304\n",
            "train loss 0.254, val loss 3.678, val accuracy 0.323\n",
            "train loss 0.099, val loss 3.864, val accuracy 0.325\n",
            "train loss 0.061, val loss 4.061, val accuracy 0.331\n",
            "train loss 0.041, val loss 4.275, val accuracy 0.341\n",
            "train loss 0.020, val loss 4.317, val accuracy 0.352\n",
            "train loss 0.013, val loss 4.469, val accuracy 0.331\n",
            "train loss 0.019, val loss 4.714, val accuracy 0.349\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 664
        },
        "id": "BmROmS2lN3Rg",
        "outputId": "936fefdf-63b4-4ace-eebc-e58e612f6021"
      },
      "source": [
        "print(\"***Model 3 : LSTM (variable-length inputs + random initialization of word-embeddings)***\")\n",
        "pred_vs_true_m3 = matrix_pred(model_variable, val_dl)\n",
        "diagnostics(model_variable, val_dl)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "***Model 3 : LSTM (variable-length inputs + random initialization of word-embeddings)***\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           A       0.29      0.18      0.22        22\n",
            "           B       0.33      0.25      0.29        12\n",
            "           C       0.00      0.00      0.00        13\n",
            "           D       0.54      0.32      0.40        22\n",
            "           E       0.19      0.15      0.17        20\n",
            "           F       0.28      0.24      0.26        29\n",
            "           G       0.43      0.16      0.23        19\n",
            "           H       0.48      0.66      0.56        65\n",
            "           I       0.15      0.11      0.13        18\n",
            "           J       0.35      0.29      0.32        21\n",
            "           K       0.34      0.56      0.42        61\n",
            "           L       0.10      0.06      0.07        17\n",
            "           M       0.34      0.41      0.37        37\n",
            "           N       0.36      0.26      0.30        19\n",
            "\n",
            "    accuracy                           0.35       375\n",
            "   macro avg       0.30      0.26      0.27       375\n",
            "weighted avg       0.33      0.35      0.33       375\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEDCAYAAADgNmW7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydeVhTV/7/3zcbAUJiAAFBtGJBitS1Fm0VrW3dqkhtqa3r17a0HbXtt1OnM+3X37TafdradsZq7egoVjsqdekotS4tdaGKiiIKgtQdFQTCFpaEJPf3RyZoSAg3yQk5Se7ree7zwE3y4XPOPTmce+65r8uwLMuCh4eHh8etCNydAA8PDw8P3xnz8PDwUAHfGfPw8PBQAN8Z8/Dw8FAA3xnz8PDwUADfGfPw8PBQAFWd8f79+9GvXz9cuHDBqTj33HMPpk6dipSUFDz++OM4efKkU/EqKyvx2muv4ZFHHsG0adOQnp6OS5cuOZzXY489hpSUFPzrX/+CwWBwOC9TPNP2zTffEItVVlbmcKyqqiq8/vrrePjhhzFt2jRMnz4d+/btszvO4MGDzX7ftm0bli5d6nBeHcV1d7w7P3/gwAGMHz8e169fd1s+Jvr164dFixa1/a7T6TB8+HC8+OKLDsX66KOP2n5fs2YN/vGPfxDJ01sQuTuBO9m1axeGDh2KrKwsvPLKKw7HkUql+OGHHwAAhw4dwrJly7BhwwaHYrEsi4ULFyI1NRWff/45AKC4uBjV1dXo06ePw3lVV1fj9ddfh1qtdrisd8ZzFlKxWJbFggULkJqais8++wwAcP36dfzyyy9Ox/Z2jhw5gvfeew9r1qxBVFSUu9NBQEAASktL0dLSAqlUipycHISHhzsUSyKRYO/evXjhhRcQHBxMOFPvgJqRcWNjI/Ly8vD+++8jKyuLWFy1Wg25XO7w548ePQqRSIRnnnmmbV98fDzuu+8+p/IKCQnBu+++i40bN8Kb7rs5evQoxGKxWX1FRUVh9uzZbsyKfo4fP47Fixfj66+/Rq9evdydThujR4/Gr7/+CgDIysrCY4895lAckUiE6dOnIyMjg2B23gU1I+Off/4Zo0aNQp8+faBUKnH27FkkJiY6FKulpQVTp06FRqNBZWWlUw2gtLQU/fv3d/jztoiOjoZer0d1dTVCQ0Pt/rypnCZefPFFTJo0yaFc7ozVs2dPfPXVVw7FKS0tRUJCgkOftZUTANTV1WHs2LFEYtOEVqvFggULsH79evTt29fd6ZgxadIkrFixAg899BBKSkrwxBNPIC8vz6FYM2fOREpKCp5//nnCWXoH1HTGWVlZmDNnDgBjA8jKynK4M77zlPvUqVP485//jF27doFhGGL50gCN0xTtWbJkCfLy8iAWi7F161anctq2bRvOnj1LOkW3IxKJMHjwYHz//fdYvHixu9MxIz4+HmVlZdi1axdGjx7tVCyZTIapU6di/fr1kEqlhDL0HqiYpqitrcXRo0exePFijB07FmvWrMHu3buJnL4PHjwYNTU1UKlUDn0+NjYWhYWFTudhjWvXrkEoFCIkJMQl8d1BbGwsioqK2n5/++23sW7dOtTU1LgxK7oRCAT44osvcObMGXz99dfuTseCsWPH4m9/+5vDUxR3MnfuXGzduhXNzc0EMvMuqOiM9+zZg6lTpyI7Oxu//PILDhw4gJ49e+LEiRNOx75w4QL0ej26devm0OeHDx8OrVaLzZs3t+0rLi52OjeVSoW3334bM2fO9KoR+/Dhw6HRaPDdd9+17WtpaXFjRp6Bv78/Vq1ahZ07dyIzM9Pd6Zjx5JNPYsGCBejXr5/Tsbp164YJEybg+++/J5CZd0HFNMWuXbuQnp5utm/cuHHYtWsXhg0bZne8O+caWZbFxx9/DKFQ6FBuDMNg+fLl+OCDD/DPf/4Tfn5+iIqKwltvveVwXjqdDkKhEFOnTsW8efMcyuvOeCZGjRplthTJHTAMg6+++goffvghVq9ejeDgYPj7+7s9L1eh0+kgkUiIxOrWrRtWr16NmTNnIjg4GA8//LBDcZqbm5GcnNz2+7x585xqZxEREW1TiCR49tlnsXHjRmLxvAWGV2jy8DhOcXExFi9ezI/0eJyGipExD48n8u9//xvffvutQ2dJPDzt4UfGPDw8PBRAxQU8Hh4eHl+H74x5eHh4KIDvjHl4eHgogO+MeXh4eCiA74x5eHh4KIDvjHl4eHgogO+MeXh4eCiA74x5eHh4KIC/A4+Hh8cryRJzFxs91lriwky40aWd8fRFVzp9z6zJ3fDI8CD4SwX4Kacea7dbqhc3f9obADByygGL115N74u0lJ448Fsljuap0Ds6EE9OjsTpwjr87/8rQPv7DQ/vNDpaJ/5Pgdn+eWkRmDw2BLn59Si+0ASdnsWAeBlGJ3XDxavNeO3d36FtNQ+2e90AAMBDT+Va5JW9JQl1Da0ovdSEuJhANDXp8czC/A7rIXtLErFYps+QjGWt7gHH6r993QOO17+1MjpSTlP9k2xjL35kqXENlgvw+owg+EkY5JzWoKJGD38/Bj27i1B4qRUnzmnN3r/qL8HE8jLlRqLtA8b6J3ksfQ2qRsZ9oiSYNEqOjVk1mJNi/3Oy+vQKwBOTo/Drb5VY/OFtp+7Nima89mIsHkkOw74DtzjFOny8Dpt33UJT8+0Hhv6YrcKNCg2eSQnH+ORg7Py5mnNuMxbm4+YtDQDgX5/eC3+pYxY5mmP5Qv2TLCMAPDslEAIBsHRNHeob7+yQNG7Li3Tdk47HFUbsWWpaauaMGQZ4IS0Y+SXNOHamyaEYjySHQSBgsOUH8yfr7txzE80teowbE8Y5VunlZrPGY+LgsToAQO+e9j2pwNQRkIDWWL5Q/yTLGBstQmy0GHtzW1DfyEIgAMQODo9ornvS8bgi9Bdy3miAmpHxY8lyRIaJsSyj0uEY8bFB0OtZnDtfb7Zf28qi9KIa8bFBzqaJUKUYAFBbp3M6lrfhC/VPsoyJMcayqOoNWPCkDP1jxBAKGFSo9MjKaUZuobaTCK7JqyNI172rj6VA5OUj4xMnTmDJkiVEk+geLELaeAW27qtDZY3e4TihwRLU1beiVWc5n1Wl0kCpkEDkxAESMMAzKWHQ6VhkH611OI634gv1T7KM4SHGEdmsCYEIkDJYt6sRGVlq6PQsnp0iwwP3cpfWe1rdd8WxZMQM540GOI2Mi4qKsHPnTuzZswdRUVEYN24c0STSnwjGrWodsg7Ud/5mG0j9hGjVWZ4OAYBWa/jvewRQ6xzr8F+cGYmE2ECszbyJ6+XkTu+9BV+of5JllEqMnYBGy2LZdw3Q/zds/vlWvPcHBVJHB+DIGS24OG49re674lh62si4w8740qVLyMrKwq5du6BUKjFp0iSwLItvv/2WaAIjhwTi3lgp3llR0dYYHaVFo4dSan00IZEI/vsex/7I7GnhSHkkFD9mV2NLluNTKd6ML9Q/yTKaRrHHi7Rmbb9Jw6KgtBUj7vVDeIgA5dWdx/Okuu+qY0nLiJcrHXbGEydOxH333YdVq1ahd2/jUrJ169aR/eNCYE6KEqeKm1HboEd4iDGdYIXx9C1AKkB4iAgNjXo0tXQ+PqhSaXFXdCDEIsbidC002A81dVrorJzGdcbM1HDMSAnH3oMq/CPjeucf8FF8of5JlrGm3tg51jVadpJ1auO+AKkAQOedqKfUfVceS08bGXc4Z7x8+XJ0794dc+bMweLFi3HkyBGQfiiIRMxAIRNiaEIA/v5mVNv2zvwIAEDyUBn+/mYUxiZxu/hQXNoAoZDBPXFyi78TGyNDye9qu3OcmRqOWanh2HdYhS/Wltn9eV/CF+qfZBkv3zReuFIGWX4NTfsarHTUrs7LBOm67+pjKZQIOG800GEWjzzyCD7//HPs3r0bSUlJyMjIaHu8/OHDh4n8cY2WxbL1lRbb6q3GNYenipuxbH0l8gq5LXX7+VAlDAYWT02NMts/ZXwP+EuF2PtrhV35zUgJw6zUcOzPqcHna8qsLprnuY0v1D/JMuafb0WzhkVSfz/4iW/vlwcyGBgnQXm1HpW13Dpj2uveHceSETCcNxro9AJeQEAApkyZgilTpqCurg4//fQT/vnPf2LkyJFO/3G9AcgtsOxouyuN0xQV1a1WX++Ii1casS3rBp6cEoX330zAkTwV7uoZgCenROHUmVq7FuNPfjgEs6dFoKJKi/zCBowZ3s3s9dp6HU4Vch9tPDoqFOHdjXN6CrkYYhGDWdMiAQAVlVrsO1Tl8bF8of5JlrFJw2LrL02YNTEQf54jx28FWgiFwOjBfhAJgU37GjnHornuScfjCiOkY8TLFbvWGSsUCkyfPh3Tp093VT5O8/fVv6P8VgtSxvfAiGEhqKtvxfe7bmDNxkt2/TeO6+MPAAgPlWDRC70sXi8oVtvVgCaN7Y5B/c1PIZ97OhoAkF9Yb1enR2sswDfqn1QZAeDQaQ3UzQaMS/JHyih/sCxw8YYOa/7TiAvX7Vt/S2vdk47HFYHQ9SPeBQsWYP/+/Vi/fj2Skoy30P/222/46KOPcOXKFURERODVV1/FpEmTOo1FzU0fd1JZo+fksbCGwQBs2lGGTTucm5NatroMy1aTm9d6bck5r48F+Eb9kyqjiVPnW3HqfKvTcWite9LxuOLq6YcdO3agpaXFbF9ZWRn+8Ic/YPHixZg6dSpycnLwv//7v4iKisLAgQNtxvOscTwPDw8PR1x5Aa+8vBxffPEF3n33XbP927dvR1xcHNLS0iCRSPDQQw/hoYcewqZNmzqNybCkl0jw8PDwUEDeQw9yfm/sD7tRX29505lcLodcbj69xbIsnnvuOYwfPx7Tp09Hv3792qYpFixYgLCwMLz99ttt71+9ejWysrKwfft2mzlQOU3Bw8PD4yz2TFNkZGRg+fLlFvsXLlyIl19+2Wzfd999B5ZlrV47U6vVuPvuu832yeVyqNWdz4l3aWdM2jWbOv+82f4B/fzx5IQQnL/UjOpaHfR6oHeUBGOHK9DcYsBrH1xBndr8dtAdK+IAkPXWkoxF0kFMor4AY52R9BnT6lkGyB5L2tqrKTdaYzmLPRfw5s6di8cff9xif/tR8dWrV7Fy5Ups3rzZahyZTIaGhgazffX19ZDJZJ3mQMXImJRrtqCkGQUllhcKCkub8UZ6JMaOkGP7PktZvTVI+mFJO3BprC+AbDlp9SyTjOUL7ZV027cHe0bG1qYjrHHixAnU1tZi2rRpZvvnz5+PyZMnIz4+HocOHTJ7rbCwEPHx8Z3GpuICHkmnrjUqVcYlQoEB3L/QJP2wJGMBdNYXQLactHqWSR9La3hTe+2K+uoIRiDgvHFl4sSJ2L9/P3744Ye2DQDee+89/PGPf0RqaipKSkqwdetWtLa24sCBA8jOzsbTTz/daWwqRsakEYsYSP0YSMQCRPeQYE5qKADg5Fnui+hJ+mG7wjXrDCTqC6C3nLQfS29ur+5sE0Ix+bGmv78//P39LfYHBwdDoVBAoVBg5cqV+PDDD7FkyRJERETggw8+6HRZG+ClnfGjDyrwwvTb/3ErqlqxbO1NFF1o5hyjMz/sgAQFRCKGk3yFZCxXQKK+AHrLSfux9Ob26s420VW3OZeUmD/M9IEHHsDOnTvtjmN3Z6xSqaBUKsEwdNzPbY3c02qUlWvhL2XQp6cU9w8IhFxm3yk3UW+ti12zzkKivgB6y0n7sfTm9urONmHP9AMN2OyM8/Pz8dlnn0GhUGD+/Pl44403UFNTA4PBgI8//hjJycldladdVNfqUF1rnHfLPd2II/kN+PSNXvCTMNi6h9sFEZJ+WFe6ZklAor4AestJ+7H05vbqzjZBiwCIKzb/dSxduhQvvvgiHnvsMcydOxfvvfcecnJysGHDBixbtqyrcnSaK9e1uFimwcTkbp2/+b9UqbRtQpn22OuHJRmrK3CkvgB6y+lpx9Kb2qs724SnWdtsdsZ6vR4jR47ExIkTERoaikGDBgEA+vbt2yXJkUQiFkBmx9Vpkn5YV7hmXY299QXQW05PPJbe0l7d2SYEIiHnjQZsdsaCO+ZcpFLzx2nTOGfcTW69UhPj/NErUoLzl1qsvm4Nkn5Y0q5ZUpCsL4DectJ6LH2hvbqzTQiEDOeNBmzOGRcXF2PIkCFgWRYajQZDhgwBYLw3W6vl/hjxziDlmn3p6XAoFUKcKWnCLZUOEjGDvr2kGDk0CC0tBqzdxv15WyT9sCRjAXTWF+ly0upZJhnLF9or6bZvD7RMP3DFZmd87hxZvWJHkHLNHjpRjzFJcoxJkkMuE4JljQvo9x6uxfZ9NaiqcY8flnQsWusLIFdOWj3LJGP5SnslGcsevGo1RVdByjWbc1KNnJPk5qBIemtJxqK1vgBy5aTVs0wylq+0V9L+Z6541ciYh4eHx1PxtM6Y9xnz8PB4JdfmP8H5vdErtrowE25QodCckRqJ2D4BiIsJRGS4FOW3NJwUmtbiORrLmvqPYYC0lChMnRCJiDApauu0yD5cidUbL1tdqG5Le+nOMpriTfyfAov9UeESjH1AiSGJMvTo7gexmMHNSi0OH6/D9j2V0Ggt/1fvXjegQ1WlI3VGMhbNbYy2WKZ4tLYxZ+HnjB0gfUZ0m7tWFuhcSiRjvfL8bQ/rpu3XjB7WKVGIjZF16GHtirxIxhqXHIzJY0OQm1+P7CO10OlZDIiXYe4TERg1TIHX3v0d2lbuBSVZZ75Q/74Qi3Qb4wyFy29tQUVnTNJdSyoWrQ5i0rEOH6/D5l230NR8e6T5Y7YKNyo0eCYlHOOTg7Hz52pOsWj24NJa/74Qi2QbswdPmzOmYhxP0l1LKhbNDmKSsUovN5t9SUwcPFYHAOjdU2rxWkfQ7MGltf59IRbJNmYPrvAZuxKbWVy5cgV5eXkW+/Py8nD16lWXJUUDtLp5u4pQpRgAUFvHfa2rt3hweboGR9qYPXiVm+KDDz6w+uwmmUyGDz74wGVJ0UBnHlalQgKRFfmJNyBggGdSwqDTscg+Wsv5cyTrzJfr3xdwtI3Z9Te8yU1RVVWFfv36Wezv168frl+/buUT3gNXD6s38uLMSCTEBuLb7eW4Xs79dJVknfly/fsCjrYxe/CqkXH7p5zeSUuLfRIZT6NFo4dYZL16aHAQu4rZ08KR8kgofsyuxpYs+9wUJOvMV+vfF3CmjdmFQMB9owCbWSQmJmLLli0W+zMzM9G/f3+XJUUDtLp5XcnM1HDMSAnH3oMq/CPD/jMfb/Hg8rgOZ9uYPTAMw3mjAZtL29566y0sXLgQO3fubOt8z549i9bWVixfvrxLEnQXxaUNSBoSjHvi5Cgoqmvbb/Kwni6ss/Fpz2NmajhmpYZj32EVvljrmEOAZJ35Wv37AiTamD3QskqCKzazDQ0NxaZNm7BgwQJERUUhKioKCxYswObNm9G9e/euytEt0OrmdQUzUsIwKzUc+3Nq8PmaModNWt7iweUhD6k2Zg+MSMh5owFON30MHz4cw4cPd1kSJN21pGLR6iAmHWvywyGYPS0CFVVa5Bc2YMxw80f91NbrcKqQm1mMZg8urfXvC7FItjF7oOXCHFeouAOPpLuWZCwaHcSkY8X18QcAhIdKsOiFXhavFxSr7fqi0OrBpbX+fSEW6TbGFYbxrGkKKjpjku5akrFodBCTjrVsdRmWrSY3f0erB5fW+veFWKTbGGc8bGTMKzR5eHi8ktqPF3J+b7c/u39BAhUjYx4eHh7S8HPGNujINZu9JalN1xcXE4imJr3DDuJX029rF4/mqYzaxcmROF1YZ1W7aHIQk4z16pfmN8tEhgoQGWI5fxUZKsTD90lw+vdW/CvL/CaaL18NIpaXKTeSsawdS3uPo+kzro7lSDxTG0udf95s/4B+/nhyQgjOX2pGda0Oej3QO0qCscMVaG4x4LUPrqBOrTf7zI4VcQBg4fqdlxbRppYsvtDUppYcndQNF682W1VLmjy/JOts+qIrNt8DALMmd8Mjw4PgLxXgp5x6rN1eY/GezZ/2JpqXszBCOlZJcIWKkTGN2kuSsW5UGXCjyvJusaeijOU8WtjKKQ7pvHxFU0kyXkFJMwpKLOc/C0ub8UZ6JMaOkGP7PsuOyhqk1ZKk68xEnygJJo2SY2NWDeakBNv9eVfl1SnetM64q6BRe0la4dgeiQgYEidGTYMB567oO/+AC/LyFU2lK+K1p1JlNI8FBnDvaEirJV1RRoYBXkgLRn5JM46daXIohqvrviO86g48T8OTFI6DYkXw92NwMF9r1zItTyqjNyMWMZD6MZCIBYjuIcGc1FAAwMmzjU7HdrVa0h4eS5YjMkyMZRkudEi4Cg8bGXtVZ9yZdnFAggIiEcPJaUAyljWG9xfDwLI4WsR9ioJ0Xq4uozfz6IMKvDD99plDRVUrlq29iaILzU7F7Qq1JFe6B4uQNl6BrfvqUFmjR3elZ83Beu0FPJVKBQAIDrZ/zqir4KpdVOs6nxYgGas9Yd0Y9I0SoeSqDqp6+zo6Tymjt5N7Wo2yci38pQz69JTi/gGBkMuc76xMasm1mTddppbkSvoTwbhVrUPWgfrO30wj3nTTB8uyWL58OTZs2ACWZcGyLIRCIWbNmoWFC7mv4esqWjR6KKUSq685onAkFas9w/sb4x6x48KdK/JyZRm9nepaHaprjdMIuacbcSS/AZ++0Qt+EgZb93C7gNeeLlNLcmDkkEDcGyvFOysqoPfQJuBpqyls/utYt24dTp48ie+//x7Hjh3D8ePHkZmZiVOnTmHdunVdlCJ3PEHhKGCAYfeIoG42oOCC/XOCnlBGX+TKdS0ulmkwMblb52+2QleqJTtDJATmpChxqrgZtQ16hIeIEB4iQqjSOHYLkAoQHiJCgJTyaQABw32jAJud8Q8//IDPPvsM0dHRbfuio6PxySefYMeOHS5Pzl6KSxsgFDK4J878nnqTdrHkd+73v5OMdSeJMSLIAwU4UWxco2ovnlBGX0UiFkBmx2oKE12tluwMiZiBQibE0IQA/P3NqLbtnfkRAIDkoTL8/c0ojE2i+wKvVz2QVKfTWZ0jDg4Ohk7n/iu97fEEhePw/sYr5fasLXZVXrym0n66ya13tolx/ugVKcH5S/Y9AccdasnO0GhZLFtfabGt3mpc83yquBnL1lcir9CxpW5dBsNw3yjA5pyxWCx26DV7oVF7SVrhCADyQAbxvYW4XK7HzWrHJuJoLiOtCkeS8V56OhxKhRBnSppwS6WDRMygby8pRg4NQkuLAWu3cZ/rJa2WJFVGvQHILbDsaE2rKSqqW62+7uq87IaSES9XbHbGxcXFGDJkiMV+lmWh1WqJJUGr9pJkLABIShBDKGBw9Kxjo2JX5OULmkqS8Q6dqMeYJDnGJMkhlwnBssYbPvYersX2fTWoquF+xkhaLUm6zkjhrrw87QKezc743DlyGj1b0Kq9JBkLAPYd12Lfcef/idFaRloVjiTj5ZxUI+ckmbl00mpJ0nXWnsoaPSePRXtcnVeHeNPSNh4eHh6PhZJVElzhfcY8PDxeScu/P+b8Xukzf+b83hUrVmDbtm2oqamBSCRCYmIiFi1ahHvuuQcAUFRUhKVLl+LcuXNQKpV49tlnMWfOnE7jdunI2JrCETBezExLicLUCZGICJOitk6L7MOVWL3xstWbDmxpLx2NZU39NyM1ErF9AhAXE4jIcCnKb2kcVns6mld7hWNkmBjTJ4UgppcfghUiiIQMKlU65BU2Ysc+FWrqra+X27EijkheptxI1BfAXeEoETP4dFEPhIeIbSoc22sqASAqXIKxDygxJFGGHt39IBYzuFmpxeHjddi+pxIareV4xJaq0tF2Mfev5RavZSyNsPqZFo0BL75veQHV9P725XSkjKZykjyWJNuY07hoZDxx4kTMnDkTCoUCWq0WGzZsQHp6Og4ePIimpiY8//zzmDFjBjIyMnDu3Dm88MILCAsLw4QJE2zGpWKa4pXnbzt1N22/ZnTqTolCbIysQ6duV8RKnxHd5mGVBTpXVaTyClWKoFSIkJuvNvPpjntQgVFDg6z6dLsiL4BsfbXnqfEKyAMduyAzLjm4zRucfaS2zRs894kIjBqmsOoNtgXpcpZc1uLXE+arEzq4S71DaC4jyTZmFy6aM+7Tp4/Z7wKBAJWVlWhoaMDPP/8MgUCA+fPnQyAQYNCgQUhLS8N3331Hf2fsC35eknmR9OnSWl8WeTrp06XdG3yrRo/fCuxbn9weWstIuo3ZhR2rKerr61Ffb+ngkMvlkMvlFvt//fVXLFq0CA0NDWAYBvPmzYNCoUBxcTESEhIguGNZXWJiIjIzMzvNwe2XG33Bz+tqNzLgmE+X1vq6ExI+XU/wBguFgJ/E8dNqWsvYFW2/QxgB5y0jIwMPP/ywxZaRkWE19JgxY3DixAnk5ubiL3/5CwYPHgwAUKvVCAoyvzNRLpdDre58BY7bR8a+4Od1RV4kfLq01teduNKnS4s3eFiCHx4YEA6hkEG9Wo/csy3Y+rMazRrnz9/dXUa3tjE75oznzp2Lxx9/3GK/tVHxnXTr1g1z5szBsGHDEBMTA5lMhupq8zOQ+vp6yGSyTnNwe2fsC35eV+RFwqdLa32ZcKVPlxZv8IUyLY4XtqCiWg9/PwYD4vzw6PBAxN8lwburVR1eeOMCDWV0axuzY864o+kILhgMBuh0Oly5cgXx8fHYvXs3DAZD21RFYWEh4uPjO41jM9t//vOfbT/v3r3b7LVly5Y5krcFXJ26XR2LJK7IK/e0Gn/9sgwfrrqOTVnVaGzW2+3TpbW+TLjSp2vyBn+7vdyt3uCl36iwO6cJJ4s1yDndgpWZdcjc34DoCDHGDQ9wKjYNZXRrG3ORm2L9+vWorDSeqalUKixZsgQSiQSDBg3CuHHjoNfrsXLlSmi1WhQUFCAzMxPPPPNMp3Ft1sKPP/7Y9vM333xj9tqhQ4fsKkBHtGj0EIusp+GIn5dULJK4Iq/qWh0KSpqQe7oRm7Kq8eX6csxNDcUT45VuzYsUJp/u6q0q4j5dmrzB1th9uBGtOhYD4/wcjkFLGd3axgQC7psdHD16FKmpqRg0aBBSUlJQVVWFdd1kiSgAACAASURBVOvWITQ0FDKZDKtXr8bBgwdx33334eWXX8aCBQswceLETuN2Kpe39rO13x2lSqXFXdGBEIsYi1MZR/y8pGKRpCvyutOny1VuTmt9WfPpAkCwwjjyN/l0Gxr1aGqxLz+avMEdoTcAtQ16BAU4NmKkqYxubWMC17gpVqxYYfP1hIQEbN682e64No/2nU9Nbf8EVVJPVPUFP29X5WWvT5fW+nKVT5c2b3BHiEWAUi5EXaP9I0bayujWNuaikbGr4GRtY1kWGo2mzeBG0tr286FKzE7rhaemRqGgqK5tv6N+XlKxSEIyr25yIWqt3GVn8ukWnud+AY/W+jL5dNsjDxTg+SdCcKq4GdnH1Lh6g3sbpNEbHOjPoLHZMpFpY4MgEjLIL7FvnpfGMrq1jVHiKeaK261tvuDnJZkXSZ8urfVF2qdLqzd46mgZ+kaLce6SFtW1ekj9GAyI9UNCjB9+v6bFvqPclynSWkZXeME5w1vb7McX/Lyk8iLp0yWZF0CvT5dWb/C5y1pEhokwcpA/Av0FYFkW5dV6ZO5vwJ7fGtFqx6GktYwAeS84Z7xpZNxV+IKfl1ReJH26AL31ZQ1Hfbq0eoNPFWtwqpjMkjNaywiQ94JzhfUwuTyv0OTh4fFKmrM3cn6v/0MzXZgJN6gYGfP4NqtWrUJhYSEKCwtRVlaGqKgo/PLLLxbv02g0+OGHH5CdnY2SkhJUVVWhe/fuGDhwIBYsWIC+ffu6IXseavGwOeMuHRlb86YCRg+qSdcXFxOIpiY9Jz+stXiOxrLmYX01/bb672ieyqj+mxyJ04V1VtV/tjzL7oxlivfqlw0W+yNDBYgMsWy0kaFCPHyfBKd/b8W/ssyNYl++GtShm9qRcvbr1w/SgG4Ii05AxbVC+ElleO4dy85YVXEBGe9PQmTMUPSOfxAyRRjqqstw+vC/odM04V//Wo03l1mfJ6ShXbz4kcoiVrBcgNdnBMFPwiDntAYVNcbbont2F6HwUitOnDNfMbLqL8HE8jLlRuJ7ZPoMybycpenAJs7vDRj9tNN/z1moGBmTVBLSqP6jNRYA3Kgy4EaV5XrWp6KM9Xa0kPvDUx3Nbd5f96NbqPHi0PoPJ6NVY32lhL8sGDPf2IGwnveY7Y8fOgUb/5aKv/3tb4DI+hMbaGwXAPDslEAIBMDSNXWob7yzV7JvLplWHapbFZoeNjKmIluSSkIa1X+0xuoIiQgYEidGTYMB565wF9U7mpupI+4M/0ClRUcMACE97kZIjziUlpZ2+Fka20VstAix0WLszW1BfSMLgcB4w4e78wLorC+7cZGbwlVQMTKmEVrVnl2hJBwUK4K/H4OD+Vq7lh65S5fIGgxorL+F0NBQl8S/E5JlTIwx6i1V9QYseFKG/jFiCAUMKlR6ZOU0I7eQ+00ttOpQ3ZmXp62moGJkTCOdqf+UCglEIm7/UWmN1RHD+4thYFkcLeI+RdFVuVmjIOffaKyvRGpqKvHY7SFZxvAQY2cxa0IgAqQM1u1qREaWGjo9i2enyPDAvRK35EUSt+Zlh1yeBmxmcePGja7KgzpoVXu6WkkY1o1B3ygRSq/poaq379quO3SJNy6exIHtH6F7VDxeeuklorGtQfRY/vfJHhoti2XfNeBYkRa/ndHi0w0NaGwxIHV0ALh2U7TqUN2ZF8sIOG80YDOLBQsWtP388ssvuzwZmqBV7elqJeHw/sbR2BE7Ltx1VW7tqbh6FjtWvYhARRhSX/wGfn6OKye5QrKMptHi8SKtmSa0ScOioLQVCpkA4VZWurg6L5K4NS8PmzO2eaTvXPV27do1lydDE1Uqbds9+e1xRO1JY6z2CBhg2D0iqJsNKLhg/2N6XJlbeyquFWLrimfh5x+EtJfXQ9YtnEjcziBZxpp6Yydkzc5WpzbuC5By64y7su7twZ15edXI2JZC09uhVe3pSiVhYowI8kABThTroOe+iKJLcruTimuF2PrVPEj8AvHkyxmQB0cRicsFkmW8fNP4D08ZZPk1NO1r4KjRpFWH6l6FppD7RgE2O2OTQnPw4MEoKSnBkCFD2n436TS9lZ8PVcJgYPHUVPMvuqNqTxpjtWd4f+PVfXvWFndVbiZuXSvCtq+ehcQvAE++vB6KEG7L4khBsoz551vRrGGR1N8PfuLb++WBDAbGSVBerUdlLbfOuCvq3hHcmRfLMJw3GnC7QhMgp+sjGYtWtaerlITyQAbxvYW4XK7HzWrH5vAcza3o2A401BgvFjerVdDrWpG7x/g0hSBlJBLuN66SqFddx9YV89DSXIdBo2fj5qWTuHnpZFucH1oAg14JgdBy7pjGdtGkYbH1lybMmhiIP8+R47cCLYRCYPRgP4iEwKZ93BWatOpQeYUmd6hYZ0xS10er+o/WWCaSEoxrXI+edWxU7ExuhUe3ouz3Y2b7fsv6EgDQ8+772zrjuuoytDQan3J8dPc/LOL8BCDsns8gEHa3eI3WdnHotAbqZgPGJfkjZZQ/WBa4eEOHNf9pxIXr3qFDdZdCk+W8FoUOqOiMSer6aFX/0RrLxL7jWuw77vzTWxzJLe2Vbzm9Lzo2Ca/9vaTD118a37H/hNZ2AQCnzrfi1Hnn/gkC9OpQ3abQ5EfGPDw8PBTgYZ0x7zPm4eHxSlQFhzi/N3jAKBdmwo0uHRl3dAo5IzUSsX0CEBcTiMhwKcpvaRzWGzIMkJYShakTIhERJkVtnRbZhyuxeuNlq4vLbakqHY018X8KzPZHhUsw9gElhiTK0KO7H8RiBjcrtTh8vA7b91RCo7X8f7h73QAA1uvM3voCjHXWPi9nckudf95if2SYGNMnhSCmlx+CFSKIhAwqVTrkFTZixz4Vaqw8SHXHijiiZSTdxkjkZorF5SklEjGDTxf1QHiIGD/l1GPt9hqz1zd/2hsAmfYKGNssifYKGNsFyWPpNJSskuAKFdMU6TOi29ypskDnUnrl+dvu1E3brxndqVOiEBsj69Cd6upY45KDMXlsCHLz65F9pBY6PYsB8TLMfSICo4Yp8Nq7v0Pbyj0xkvVFMrdQpQhKhQi5+WpU1xrXKveOkmDcgwqMGhqE1z64gjo1twXMJMtIOh7p3O7kqfEKyAMdW/fqC+3VHvg5Yweg0Z1KMtbh43XYvOsWmppvj05+zFbhRoUGz6SEY3xyMHb+XM2xlGT9zyRzKyhpRkGJ5UWawtJmvJEeibEj5Ni+r8bKJy0hWUbS8UjnZqJPlASTRsmxMasGc1KC7fusj7RXe/C01RRU/Oug0Z1KMlbp5Wazhm3i4LE6AEDvnlLOsQCy/mfSuVmjUmVcohUYwP1LSLKMpOORzg0wnlG/kBaM/JJmHDtjXa5vC19pr/bgabdDUzEyJoWneYNDlcbbrmrr7PdAuBpnchOLGEj9GEjEAkT3kGBOqtEzfPIs95sYfI3HkuWIDBNjWUalQ5/39fZqFW+aM96/fz8qKiowc6bxyalpaWlQqYzP8frTn/6ECRMmuD5DO+jMnTogQQGRiOEkJiEZyxoCBngmJQw6HYvso7UOxXAVzub26IMKvDD99kisoqoVy9beRNGFZpJpeg3dg0VIG6/A1n11qKzRo7vS/tN4X26vHWFg6HBOcMVmZ7x69Wp8/vnnbb9rtVp8//33aG5uxptvvkldZ8zVnarWdX4RiWQsa7w4MxIJsYFYm3kT18vdcxrXEc7mlntajbJyLfylDPr0lOL+AYGQyzzri9GVpD8RjFvVOmQdqO/8zR3gy+21I2iZfuCKzc64tbUVPXr0aPt96NChUCqVUCqVaG6mb5TTotFDKbX+dARHvMGkYrVn9rRwpDwSih+zq7Ely7HTUldBIrfqWh2qa42nsrmnG3EkvwGfvtELfhIGW/dwu4DnK4wcEoh7Y6V4Z0WFmdPYXny1vdrCqy7g1deb/6f+61//2vazabqCJjzBGzwzNRwzUsKx96AK/8i43vkHuhBX5XbluhYXyzSYmNyNWExvQCQE5qQocaq4GbUNeoSHiBAeIkKo0jhGCpAKEB4iQoC0807FF9trZ3jaBTybWQwYMABbtmyx2L9p0yYMGDDAZUk5Cu3e4Jmp4ZiVGo59h1X4Ym3X3qffGa7OTSIWQGbHagpfQCJmoJAJMTQhAH9/M6pte2d+BAAgeagMf38zCmOTOr/45mvtlQtepdB86623sGDBAuzcuRP9+/cHABQWFkKr1eKrr77qkgTt4edDlZid1gtPTY1CQVFd235HvcGkYgHAjJQwzEoNx/6cGny+psyltip7IZVbN7kQtVbuskuM80evSAkKz9M3teVONFoWy9ZbnvbLAwV4/okQnCpuRvYxNa7e6Fzg5EvtlStedQEvJCQEmzZtwpEjR/D7778DAEaPHo0RI0YQTYJGdyrJWJMfDsHsaRGoqNIiv7ABY4abn67X1utwqpD7yIWk/5lkbi89HQ6lQogzJU24pdJBImbQt5cUI4cGoaXFgLXbuM83kiwj6XikYukNQG6B5Zpi02qKiupWq69bw1faqz142pwxp3XGI0aMIN4B3wmt7lRSseL6+AMAwkMlWPRCL4vXC4rVdjVukvVFMrdDJ+oxJkmOMUlyyGVCsKzxho+9h2uxfV8Nqmq4r08lWUbS8UjnRgpfaK/2QMtcMFeouOmDVncqqVjLVpdh2Wpyc24k64tkbjkn1cg5SeaZZiTLSDoe6dzaU1mj5yQVao8vtFd78LSRMa/Q5OHh8UouXfid83v79L3bhZlwg4qRMQ8PDw9pPG1kTIXPOHtLUptiLy4mEE1Neod9xq+m39YIHs1TGTWCkyNxurDOqkbQls/YW2KZ4s39a7nlCwAylkag5LIWv54wv1ikMwDHzrZYfb+1vBwtZ5a4n9m+ez5+A73Sn4YoKBCXV2xA4avvtr02eOPn0Le0QF10AdrqGkiUCkQ8OQHK+wcCIN/G2rt+56VFtOkliy80teklRyd1w8WrzVb1kiY39dvrzR+tFCgF0ieJEOQP5JUacKsGCFMCQ2MFqKwD1uzWobXd4pQlc8QdltPeMpo+Q6KMpnKSzMtZDHR40DhDxcjY2xWatMa6k1s1evxWYNnx2gOJ3OSDE3DXK3NR/OYnSPjkTYvXT818zWLfpb9nYOSxbZAPiIeAAQxW/hmRamMk9ZKjEgVQyhhkHtTh7OXbSV+9xSItWYQRCQIcPMP9rjkay0gyL3thPawzpiJbb1do0hqrPUIh4Cdx/NTO6dwEAgz4+l1U7jmE8u37OP9dVq9Hy3XjOlqhlTvQAHJtjKResk+EAFoda9YRA0DhZRatOhaD77bv60ljGUnmZS8sGM4bDVAxMiYFrQpNWmPdybAEPzwwIBxCIYN6tR65Z1uw9Wc1mjXcr+86m1vMq/+DwH4xyHvqlU7/llipACMUQBKqRMQTE9F9vPEZZq12PIGCJI7oJYVCwJq3hwXQqgeCgxgE+AFNlHh5PE2h6apO9pNPPsGvv/6KmzdvIiAgAMnJyfjTn/4EpVLZ9p6ioiIsXboU586dg1KpxLPPPos5c+bYjEvFyJgUnWkElQoJRB2MnHwxlokLZVrs+FWN5Ztr8c3WWhRd0uLR4YH4v+eC7RopO5Ob/109Efv2yyh9bwWar3TuQBhTtAeP3jyK0Wd2I/b//oDybXs550kaR/WSlbUsAvwYRCjN90cogQA/Yz0pAklm6jieqNB01chYKBTik08+QW5uLn744QeUl5fjzTdvT6mp1Wo8//zzGDlyJI4dO4YvvvgCy5cvx08//WQzrleNjGlVaNIay8TSb8ylTzmnW3CtQoe0R4IwbngAdh7kJoV3Jrd7v3oHTZeu4dIXazn9rby0hRBI/SCNCkePJyZA4O/H6XOuwFG95JFzBsRHM0gbLcJPx/W4Vcuiu4LBxGFC6PQsRELmv+If968+9UiFpotGxn/84x/bfg4JCcHs2bPx+uuvt+3bu3cvBAIB5s+fD4FAgEGDBiEtLQ3fffedTe2wzc743XffBWNDorF48WJ7yuByaFVo0hrLFrsPNyJ1jAwD4/w4d8aO5hY1IwWhjzyII2NngdVxOwVWHT7R9nNZxjYM+vYzAIAsUAh1o2POXkdwRi959RaLzEN6TBomxKyHjV9FvYHFyVIDAqUMEnozHT6JuSvxVIWmgeV+4l9fX29hqQQAuVwOuVxu5RO3OXLkCOLj49t+Ly4uRkJCAgSC238/MTERmZmZNuPY7IwTExPbfv7HP/6Bl19+2WYwd1Ol0uKu6ECIRYzFqbIjCk1vj2ULvQGobdAjKIB7g3Y0t3s++Qtu7T4ATXklAvoab7+VRoYDAESKIAT07QVtVQ10dQ0d/u3r3+5A1NOTkXx/MH7M7poOg4ResugKi3NXdQjvZjSsVdezaGwB0icJoTewUHVc5C7BoxWadoyMMzIysHz5cov9CxcutNnv/fjjj8jMzMSGDRva9qnVagQFmV8fkcvlUKtt351qszN+/PHHzZK983caKS5tQNKQYNwTJzczV5k0gqcL62x82vdi2UIsApRyIS6UtXb+Zidz8wsLQfhjDyH8sYcsXus5cyp6zpyKc298jIuf/6vDv22apgiSdc3MG0m9JMsC5TWAaTpCJgV6BDO4XMFarDPuSjxeoWlHZzx37lyr/ZutUXFWVhbeeecdrFy5ss1qCQAymQzV1eZL/+rr6yGTyWzmwHnYY2u6ghZ+PlQJg4HFU1OjzPY7qtD09lgAEOhv/bhOGxsEkZBBfgn3+UFHc8ub/orFdmbhOwCAWz8dRN70V1Cx6xcIA/whDAywDCAQ4K4/GJ/TWFRKxo1hC1fqJRkAE+8XgmGAgwXOTzc5ijcoNFmW4bzJ5XL07NnTYuuoM87MzMSSJUvw9ddfY/jw4WavxcfHo6ioCAbD7eNXWFhoNpVhDSou4Hm7QpPWWAAwdbQMfaPFOHdJi+paPaR+DAbE+iEhxg+/X9Ni31HuT3R2NLfybXss9vn3NnboTRevtr0uHxiP4T9vQPm2PVCXXEJrTS2kkeGInD4ZsvgYAMCZYuvn9aTaGEm9pERkvAOv+KoBNWoWUgmDxLsYRIUKsP+kHpcr7OsBaSwjybzsxeCiC3jr16/HV199hTVr1uDee++1eH3cuHH49NNPsXLlSqSnp6O4uBiZmZl45513bMa12RkPHjy4bUTc0tKCIUOGAABYlgXDMDh58qSDxTHH2xWaNMc6d1mLyDARRg7yR6C/ACzLorxaj8z9DdjzWyNa7VxSSjK39jSXVeD6xh8QPPI+hE99BKKgQOjq1KjLL0LpBysweP2nHX6WVBsjqZfUG4CKGhb39hFAFgC06oAb1SzW79fhwg37K4vGMpLMy17suYBnD++//z5EIpHFuuGsrCxERkZCJpNh9erVWLJkCVatWgWlUokFCxZg4sSJNuPa7IxPnTrlfOYc8HaFJs2xThVrcKqY3FIlUrk1X7lu4axora4x81S0x1ZnTKqNkdRL6g3A94fITQrTWEbA+xSaJSUlnb4nISEBmzdvtisuFdMUPDw8PKRhWfqvc90J7zPm4eHxSo6XcL9TcFg/9z+5vEtHxqnzz1vsiwwTY/qkEMT08kOwQgSRkEGlSoe8wkbs2KdCjZUHXO5YEQfAul6SYYC0lChMnRCJiDApauu0yD5cidUbL1u94cCWqtLRWNY0gjNSIxHbJwBxMYGIDJei/JaGk8KRRCxTPJKxOlJoOlJnr35pedFtQpIEE4d3fFedXs/ij8vN5y2/fDXIQgcJAFHhEox9QIkhiTL06O4HsZjBzUotDh+vw/Y9lVZvrDBpL0m2i/bt391t35QbrbGcxdNGxm6fpghViqBUiJCbr0Z1rQ56PdA7SoJxDyowamgQXvvgCurU3OfVXnn+tk930/ZrRp/ulCjExsg6dP12Raz0GdFtTldZoHPVTmssgFydFVzQoarW8osaGSrEw/dJcPYS9yuL45KD2/y82Udq2/y8c5+IwKhhig79vB1Bqoy+0vZJxrIH9y0MdAy3d8YFJc0oKLG8WFBY2ow30iMxdoQc2/fVcIpFszeYpNOV1lgk6+xGlQE3qiy/Tk9FGfM7Wsj9ZhSSfl6SZfSFtu8q/zYXXLWawlVQm22lyjjyCQzg3jnQ7A0m6XSlNZYrXcuAcV3ukDgxahoMOHeF+4iRpJ/X1WUEvKvtd0V9dYQ9N33QgNtHxibEIgZSPwYSsQDRPSSYkxoKADh5lvtNB57gDfZmXF1ng2JF8PdjcDBfS+TU1hE/ryvK6M1t353fI1qk8VyhpjN+9EEFXph++79kRVUrlq29iaILzZxjdObTHZCggEjEcBLpkIzlK7i6zob3F8PAsjhaxH2KoiMc9fO6ooze3Pbd+T2y9vgtmqGmM849rUZZuRb+UgZ9ekpx/4BAyGX2zV/S7g32dlxZZ2HdGPSNEqHkqg6qeue/ZY76eV1RRm9u++78HnnVyPjO26HvhPTt0ABQXatDda3xdDH3dCOO5Dfg0zd6wU/CYOsebhcxPNEb7E24ss6G9zfGPWLHhbuOcMbP64oyenPbd+f3iJa5YK5QcTu0Na5c1+JimQYTk7txbpCe6A32JlxVZwIGGHaPCOpmAwouOPf8NWf9vF3RLryp7bvze6T3sM6Y2tUUACARCyCz44pycWkDhEIG98SZS0lMPt2S37nLTUjG8hVcVWeJMSLIAwU4UWxci+soJPy8XdUuvKXtu/N75GmrKdzeGXeTW29wiXH+6BUpwflLLZxj0ewN9gVcVWfD+xtXPdiztrg9pPy8JMvoC23fnd8jluW+0YDbL+C99HQ4lAohzpQ04ZZKB4mYQd9eUowcGoSWFgPWbuM+p0ezN5ik05XWWKTrDADkgQziewtxuVyPm9WOzS2S9POSLKMvtH1XtAmueNUFvK7g0Il6jEmSY0ySHHKZECxrXPS+93Attu+rQVWNfXOEtHqDSTpdaY0FkPcZJyWIIRQwOHrW8VExaT8vqTL6Stt3pePaFvzSNjvJOalGzkly80a0eoNJOl1pjQWQrTMA2Hdci33HtU7FIO3nJVVGX2n7pNsE97/rWSNjXqHJw8Pjlew6yf3MYvIQt49L3T8y5uHh4XEFnjbM7NLOuCMH7qvptxV7R/NURsXe5EicLqyzqtgzuU7bu2vnpUW0qRKLLzS1qRJHJ3XDxavNVlWJtry1juZFMpY1B3H2lqQ27WVcTCCamvQO+4wdjUXyWFpzEDt6LEm3sRc/UlnECpYL8PqMIPhJGOSc1qCiRg9/PwY9u4tQeKkVJ86ZT6us+kuwMYd23ubIUAEiQywXNJk0oad/b8W/ssxXVHz5qtHlQKKNmcpJaxtzFlqWrHHF7SNjkoo9WlWJvqLj9IVjCQDPTgmEQAAsXVOH+sY7ezj7DHgkNaG+0sbswdMu4Ll9nTFJxR6tqkRf0XH6wrGMjRYhNlqMvbktqG9kIRAAYoJDGkc1ob7SxuyBX2dsJ12h2HO3KtFXdJy+cCwTY4x/X1VvwIInZegfY1x6V6HSIyunGbmFzq38cFQT6ittzB487XboDjvjjiRBACCRSNCrVy+89tprGDFihFMJuFqxR4Mq0Vd0nL5wLMNDjKfYsyYE4laNHut2NUIkBB65X4pnp8ggFKjx2xnHO2RHNaG+0sbsgZYRL1c67IxtSYL0ej1KS0uxaNEi7Nq1y6kEXK3Yo0GV6Cs6Tp84lhLjAEWjZbHsuwbo/xs2/3wr3vuDAqmjA3DkjBaO9APOaEJ9pY3Zg6d1xg7NGQuFQsTHx2PWrFlOJ9Ci0UMssp6Gs4o9Z1WJpPJyZRlpwheOpWnkebxI29YRA0CThkVBaSsUMgHCrayQ4IIzmlBfaWP2YGAZzhsNOHUB7+mnn3Y6gSqVts2J0B5nFHskVImk8nJVGWnDF45lTb2xQ6trtOzY6tTGfQFS+79WzmpCfaWN2YOnXcBz+2oKVyj2aFMl+oqO0xeO5eWbxo5SGWT51THta7DSUXeGs5pQX2lj9qA3cN9owO2dMWnFHo2qRF/RcfrCscw/34pmDYuk/n7wE9/eLw9kMDBOgvJqPSpr7f92O6sJ9ZU2Zg+e5jN2+9I2koo9WlWJvqLj9IVj2aRhsfWXJsyaGIg/z5HjtwIthEJg9GA/iITApn3cn+hsgoQm1FfamD3QMv3AFbd3xgA5xR6tqkTSsXxBoUnzsTx0WgN1swHjkvyRMsofLAtcvKHDmv804sJ1++d7SWhCAd9pY1zxtDvwqOiMSSn2aFUlko7lCwpNmo8lAJw634pT551/OCpARhMK+E4b4wo/Mubh4eGhAE/rjHmfMQ8Pj1fyz/3c35v+iOvy4AoVCk2GAdJSojB1QiQiwqSordMi+3AlVm+8bHWhekcKzahwCcY+oMSQRBl6dPeDWMzgZqUWh4/XYfueSmi0lv93bCk0Hc2LZKzU+efN9keGiTF9UghievkhWCGCSMigUqVDXmEjduxToabe+rqoHSviiORlyo3ksWyvlgSACUkSTBzuZ/VvAIBez+KPy83njL98NciqwhEAZqRGIrZPAOJiAhEZLkX5LY1NjaNJ4UiyjemryyGQKQCBEIZ6FVovFkGTux9sY73FZ0xIBo1EwHjjev66v/8Zilc+BmBdrWpvGQHg6YfzsWN/ISrLCtGgKkOQMhKz3vrF6nuvnc/BxTN7UVlWCFX5eeh1WqS8lIGovsa6+t8UhmgbcxYDJUvWuELFNMUrz9/2sG7afs3oYZ0ShdgYWYceVmuMSw5uc+BmH6ltc+DOfSICo4YprDpwuyIvkrFClSIoFSLk5qtRXWtck9o7SoJxDyowamgQXvvgCurU3BeqkiwjyXgFF3SosrJEzOT6PXvJvgtl6TOi25y6skDHm70zbaz1/GkYGmoBgwHC7pHwG/ggJPFD0bDuQ7BNlhcjGZkC/mOmgtW0gPHr3FLnSBmXLVsGvwAFukf1h7bZ8p/iV4h1JQAAGDtJREFUnZSe3InS/CwER8RCGdYXVTe4zQWTbmNc8bRzfrd3xr7gwCUZq6CkGQUllhdoCkub8UZ6JMaOkGP7vpouz4t0PJKuX4CcU9eZNtZycKfZ77prvyMw9TlIEodDc8zynNr/0aegr6mCoeomJIn3d5qbI2Xcv38/tp7qCQDY9OkU6LQdL827f+JrGP3kUghFEuT/uoZTZ0y6jdmDp3XGbr/pwxccuKRds9aoVBlHioEB3DsZ0nm5upyOun4Bck5dkm3MUG98iggjDbB4TRw7AOK770Xz3k0Ay+1825EyRkdHc36vTBEOoUhiV/yuaPsdYWC5bzTg9pGxLzhwXVFGsYiB1I+BRCxAdA8J5qSGAgBOnuV+0wHpvFx9LB11/XYFnNqYUARG4geIxBCGREA6ZioAoPViofn7JFL4P/oUtPmHob95BRg8ylVpuxx3epbtW5vg/rvwbHbGN2/eRI8ePay+lp2djYceesjpBHzBgeuKMj76oAIvTL89qqioasWytTdRdKGZcwzSebn6WDrq+nU1XNuYZOADCHj0qbbf9bVVaNy5DvqyC2bv8x8zFWAYNB/4j8ty7irc6Vl2xPHhTmx2xvPmzcPq1avRs2dPs/3ff/89vv76ayKdsU84cF1QxtzTapSVa+EvZdCnpxT3DwiEXGbfPCjpvFx5LJ1x/boarm2stbQA6uoKMBI/CMN7QnT3vWD8ZWbvEUbFQDLoQTTtzAC0LR1E8hzc6Vmm7eypM2x2xm+++Saee+45rFq1CnfddRcAYNWqVdi1axc2bNhAJIEWjR5KqfV5KHc7cEnl5YoyVtfqUF1rPCXOPd2II/kN+PSNXvCTMNi6h9sFPNJ5ufJYOuP6dSX2tDG2oRa6BuPIubW0AIKSfATN+RMYsQSao3sBgRABE56B7nIJWs/ldUX6LseVbaIzaJkL5orNC3ijR4/GO++8g/T0dJw/fx7vv/8+srOzsWHDBkRERBBJwBccuF3hmr1yXYuLZRpMTO7W+ZtdlJeryums69dVONvGDJU3oL9VBr//zgn7DUmGIDgcmuO/QNAttG2DxHhRUKAIIZp/V+BOz7KrfMZZWVmYMWMGhgwZgn79+lm8XlRUhKeffhoDBw7EmDFjsH79ek5xO11NMWLECHz44YeYPXs2rl27hoyMDCgUCvuyt4EvOHC7yjUrEQsgs2M1Bem8XFVOZ12/roBEGwMARiRuW00hUASDEQgge2oB5C++07ZJ+g0CAATNfYNI7l2JOz3LrIHlvNmDXC7HjBkz8NZbb1m8plar8fzzz2PkyJE4duwYvvjiCyxfvhw//fRTp3FtdsaDBw/GkCFDkJ6ejsbGRhw9ehQjRoxo208CX3DgkozVTW69s02M80evSAnOX+I+z0i67l3l1HXW9UsaUm1M1CsWgtBI6G5cBgBoCo6iccdqi631ivEuzKYfyUwNdiXu9CzbI5evr69HWVmZxVZfb3l35KhRozB58mSrywL37t0LgUCA+fPnw8/PD4MGDUJaWhq+++67TvO1OWds66GkpPAFBy7JWC89HQ6lQogzJU24pdJBImbQt5cUI4cGoaXFgLXbuM+Nk3bgko4HkHH9AuScus60MWnyFBjqVIBIDFFENMT3DAW0LWjJ3gYAMFReh6HScrpD3DcRAND6+xniZdyxYwfyfrkBAGhpVEGvb0Xe/pUAAJkyEv2GTm17b/WNElwuMt4qffPySQDA+bz/oPyS8eeGh2ZbxHdFm+CKwY4Rb0ZGBpYvX26xf+HChXj55Zc5xykuLkZCQgIEgtvj3MTERGRmZnb6WbevMwZ8w4FLKtahE/UYkyTHmCQ55DIhWNZ4w8few7XYvq8GVTX2zamSLKMr4pFy/ZJy6jrTxiT97wcTIANYFoZ6FbT5OWjJ3Q+2gdsF185wpIxbt27FsWPHzPYd2/MlACAyZphZZ1x5vbDtNRPFx7e2/Vz3RorVvEi3Ca7YE3vu3Ll4/PHHLfbL5XIr7+4YtVqNoCDztdNyuRxqdef9DhWdsS84cEnFyjmpRs5JcvNspD2/pOORcv2Scuo608bqV/4/hz7X9OMGTlMUjpTx22+/xRf/4dZrxQ+bhvhh0zp8vWdPBsAFi/2k2wRX7OmM5XK53R2vNWQyGaqrzW+Hr6+vh0wm6+ATt+EVmjw8PF7Ju//mfpb4/56xf1yam5uLOXPmoKSkpG3ftm3b8Pnnn+PAgQNtUxWffvopCgoKOl1V4XY3BQ8PD48rYA3cN3vQ6/XQaDRobTVOnWk0Gmg0GhgMBowbNw56vR4rV66EVqtFQUEBMjMz8cwzz3Qat0tHxh05cF9Nv63YO5qnMir2JkfidGGdVcWeLW+wt8Wy5q3N3pLUpkqMiwlEU5O+U29t9pYkCzcvAMxLi2hTQhZfaGpTQo5O6oaLV5utKiF3rxtANC+SsUi3MRK5mdzIf/7G/FZ1BsCLUyToHS5AXqkeVysMkIiAgXeL0CtMgF/zW7H7mPno7uMX/InlZfoMrbGc5e313K8zLJkj7vxN/2Xbtm148803LfavX78eSUlJKCoqwpIlS3Du3DkolUo899xzmDNnTqdx3T5nTKuqktZYADkdJEBWO0oyL5KxaK3/6DAB+vQQ4tAZHXYdud1xHCnS4/Wn/JB0j8iiM+6KvGiOZQ+uGmdOmzYN06Z1PHeekJCAzZs32x3X7dMUtKoqaY0FkNNBAmSVkCTzIhmL1vo33SVc32jeaegNQFMLoLXzZkNa659kLHvgFZp2QquqktZYXYUj2lFaobX+r90yoEnDYvRAEWoaDLh2i4VYBAyNEyIqlMH2w3Tc5OKp2Htnnbtxe2dMq6qS1lhdgaPaUVqhtf6btUDGHi2eTBZj1qO3n/fXomXx7T4tiq542EPcKMPT1ok5PE2xbt06IglwVezxsboOkxLy2+3ldmlHaYXm+te2siivYXHgdCvW79Xg+wNaVNezeOZhCWKj6GkTnojBwHLeaMDtnXGLRg+xyHoajqgqvT2Wq3FGO0ortNZ/hJLB/Kl+KC3T48dcHQovG3C8RI+VP2jQ0MRiWrIYjPsfQOGxGPQs540GHO6MSV2ppFVVSWssV+KsEpJWaK3/kQNEEIsYnLlorqJr1QPFVw0IDhJAKeN7Y0cxsCznjQYc7owZQv+yaVVV0hrLVZBSQtIIrfWvCDB+h6x9lUyeGQE/U+EwLMty3miAk0Kz/TZ48GDcukXGtkSrqpLWWK6AlBKSVmit/4paY0Xf18/8OrpUAvTvLURTC4tqyh4x5Ul42pyxVyk0fSEWQE4HCZDVjpLMi2QsWuv/8BkdhsQKMeF+ESKCGVwuNyDAj8H98ULIAxlsP2zfU7BprX+SsezB0wYVbl/aBtCpqqQ5FikdJEBWO0oyL5KxADrrv1bNYvl2DR4ZKsLdkUIM7CtEqw64UW3ArqMaFF6276IirfVP+lhyRa+n46I4V6jojGlUVdIci5QOEiCrHSWZF8lYAL31r2pgseXXVgDO3+BBa/2TPpZc4W/64OHh4aEAT+uMeZ8xDw+PV/KHT7jfPbryT9yfqu4qunRk3JHekGGAtJQoTJ0QiYgwKWrrtMg+XInVGy9bXYxvS1VJQyxrGsEZqZGI7ROAuJhARIZLUX5Lw0m72F57GRUuwdgHlBiSKEOP7n4QixncrNTi8PE6bN9TCY3W+v/W3esGECmjqZwkj2V7taQJiQh4MFGEgXcLESxjoDMAVbUG5BbrkXfe8jHRH7/gb7XuAcfrn2S7sKZ0DJQCDw0UIK6nAIFSQN1iXGOcnW9Ai5WZC5PqkUQbA4CnH87HvsNncf1SEVSVZVCGRuIvX+y3+t4tq95C3qEdVl+b+crn+OuCSUTbmLN42siYimmKV56/7ZrdtP2a0TU7JQqxMTKrrllPjJU+I7rN6SoLdLzaxyUHt/mHs4/UtvmH5z4RgVHDFFb9w7YgWUaS8RgAz0687fr97ext1+9TYyQI62bp+rUFqfoHyJUxUAqkTxIhyB/IKzXgVg0QpgTuixOgd7gAa3br0Gr5P6dDHCnjsmXLECBTIPKuBDQ3WT4J2RrTX/rIYl90zL0dvp90G+OKp530u70zptUbTKsDl6R/mHQZScaj1fVLsoyjEo132GUe1OHs5dsdx9VbLNKSRRiRIMDBM9xXBDhSxv379yOv3LjMbNlfUqBtaer0M0NGWn/wqDVItzF78LTVFG6/v4dWbzCtDlyS/mHSZSQZj1bXL8ky9okQQKtjzTpiACi8zKJVx2Lw3fZ9PR0pY3R0tN2fYVkWLU1qGAydd3ak25g9sAaW80YDbh8Z0+oNptWB2xGO+IdJl5FkPFpdvyTLKBQCOivTECyMforgIAYBfkATZeK8t9Pvh6alEUKRGH3i78P4J19Br7sHWn2vO79HtHSyXHF7Z0yrN5hWB641HPUPky4jyXi0un5JlrGylkX33gJEKIHymtv7I5RAgJ9RWKEIpKczDlKEYuSEuejZJwESP3/cuFqCnJ++xdfvzsa8P30NDBtl8Rl3fo9oEQBxxWZn/NJLL9n88Ndff+10Alxds2prQwgPieVqTP7htZk37fIPky4j6Xgm12/RlVZcqTDeKjyivwjPPCzB+j1alF7v+g6ZZBmPnDMgPppB2mgRfjqux61aFt0VDCYOE0KnZyESMv81zdHRqUx8+o9mv/e/7xEMfuAxfPF/T2D72qX407x9Fp9x5/fIq0bG+fn56NGjBx577DEMHDjQJVcnWzR6KE0ThO1wxBtMYyxX4ox/mHQZScYzuX53HmlF7rnbX9T83/V4Lc0P05LF+NsmTZf7B0iW8eotFpmH9Jg0TIhZDxu/inoDi5OlBgRKGST0ZjpcqkgLoRF3YcD945F3aAcuXbpk8bo7v0detZoiJycHOTk5yMrKwq5duzB69GhMnjwZsbGxxBKoUmlxV3QgxCLG4lTGEW8wjbFchbP+YdJlJBmvM9fvg4kiKGUMVA1dewxI11nRFRbnruoQ3s2o9KyuZ9HYAqRPEkJvYKFqIF0C8ii7G214NTU1Fq+583uk72BETis2L9cKhUIkJyfj448/xpYtW9C7d2/Mnj0bGzZsIJYArd5gWh24Jkj4h0mXkWQ8Wl2/rmgXLGucM756y9gRy6RAj2AGlytYu9YZu4vq8isAgNDQUIvX3Pk98iqfMQBotVrs3bsXixYtwsaNGzF79mw8+uijxBKg1RtMqwMXIOcfJl1GkvFodf26ul0wACbeLwTDAAcL6BnZaVua0Kq1vB5x/XIRCo7tQVhkDHr1srT+ufN7xBoMnDcasDlN8cYbb6C0tBTJyclYuHAh4uLiiCdAqzeYVgcuSf8w6TKSjEer65dkGSUi4x14xVcNqFGzkEoYJN71/9s795i2rjuOf69tbPzkmdDBmgddaZMVb8PpSLcszbJ2W5OKJt0SUDQRLSNdtiSaJo3+OW1MmpKlSqIlKtJExgqplC24kEDXB4m6taVrSIBAIYVkFEKzNYQABhvM69r7w7OL8QP7ci45Jr+PdCW41z787k/Wz4dzz/kcARmpClxoFtHbH92XjZR7rKmpwcXLniGusdFhiDPTuFjjeTCflJruW+Bxt/8m/vyHn+LL676DlLSVUGu0+KyvC1fefQ0KhRLP/+S3QWNi/RmLBl6k8ZESthifP38eWq0Wvb29qKys9J13u90QBAHNzc1MguDVG8yjA5elfxhge48s2+PZ9cvqHkUX0D/sRvZqBQw6/P/+3Ki4MIPu/0affCn3aLVa0djY6Hfu7ao/AgAyH33cV4yNCan40mNPoPtaI1oa6jA9PQljYirMuc/g23l7sTw9M2RcrD9jkcLL8EOkhC3GnZ2dixIEr95gHh24LP3DANt7ZN0er65fVvcouoCq99gNCku5x8rKStRcnj8GY+IyFPzssJSwmH/GIv67MfYA754v+iAIgpADlzu2ijH5jAmCWJJsP3Aj4tdWn2Q3XVcqi9ozDuWafedvuT71X1amHuPjomTX7C/2fq7r+7BpyKPrezYdrR0jQXV94XzGPLQVLGfR5sv7Hl7bCuVGlpIznj9j+b+6GfLvefnRs4l4ar0R2ngF3mwYRXm1/9zdv760EkCg5/rHOx7wqVU7u8d9atUncxPxSZ8zpFr1jb+Yse3n1/3OmR/R4offT8H1HicGbTMQRWBlhhqb1yfAOeHCL39/EyOOwKGNmpezsPvXt4Pe1yslD6Crdwr/uOJvhZtxAY3tE0Ffv1CW1Aq8xYJHvSGvbQHs8sVzW7zmTE4l5OoMNbZ8y4RXXx9GYV5yVO9lqVZt63KirStwfLfjhhMv7k3H5idMqK4PXOAxH3eGRXzQFlh45SLW/um/5wpNgE+9Ia9tAezyxXNbvOZMLiWkIAAv7EjG1S4nGj+a3yk8F5Zq1VAMDHmMgHqd9C9ZpRLQqIOs5JEBl8sV8cEDXPSMWcGrQjPWdJw8wGvO5Ipr60YT0pfH4egr0flF5kOKWtVLnEpAvEaAOk6BB7+gRuE2zwq75vYxSbE8vlaDb5jToFQKGHWIuNQ+AetFB5yT8vRgXWIMLF+cxZIqxrwqNGNJx8kLvOZMjriWJauw43sJsNaPYGBYxLIk6T3P2UhVq3p5+psJeCH/855+/91pHC3/DNe6g+9ZGI7uW1O43DGB/kERWo0Ac5YGT6/X49FVavyubEgWIRKNGd9DeFVoxpKOkxd4zZkcce39QTLuDM7g9X9GtgddpEhVq3q51OrArdtT0MYLWP3FeHzdrIfJIO2LouRPQ36/N7RO4NP+Gex4yojvrteh9l1pve1wLKlifPLkyZDXBEHA/v37mQe0EHhVaMaKjpMneM0Z67g25OiR/XA8fvNyP1hu2bYQtaqXQdsMBm2e4Y1LrWP411U7XnpxBTRqAda3on+AN5c33h/Dtk0GfCVLI0sxjrV5xmEf4Ol0uoBDEARYrVaUlZUtVowRc3doyrcmfy5SFJo8tnW/wGvOWMalUgKFeUlo6XTCZheRlqJCWooKqUmePpIuXoG0FBV08dE98FqoWjUUN/8zhU9uTeKZjYnzvzgCRBdgs4sw6uSZR7Ck9sDbs2eP72eHw4GKigpYrVZs2bLF7xovdN6wIzcnGWuyTGi7NuI779X1tXaMhHl3bLR1v8BrzljGpY4TkGBQwrJWB8taXcD1jRYDNloMqKwdRl2EQxgs1KrhUMcpYFjAbIrZxKmAJJMS3bfk2c+QFxtbpMz7lWSz2XDs2DHk5eVBFEVUV1ejuLgYKSkpixFfVPCq0ORZx8krvOaMZVyTU24crRgIOMqsnvnALZ1OHK0YQFNHZFPdWKlVE03Bi+1jWVqsSFfjek90c4X12uA9++c3G6FSCrjaJc8mfy5RjPjggbA948OHD6O+vh47d+5EbW0t9Hq9LEHwqDfktS2AXb54bovXnLGMS3QBl9oCC613NkX/4HTQ68FgqVbdV5CGpAQlPuoax52hGajjBDy0Ih4bLEZMTLhQ/lp0Y9DPPWnAQw/G4eOeKQzaRMRrBJgf1mBtpgb//nQK9R+yHy8GlphCs7y8HGq1GqWlpX6bj7JWaPKoN+S5LZb54rUtgN+c3SslZDhYqlXfuzKKTbkmbMo1wWRQwu32LPh4+30bquuHcXc4ujnLH/dOIX25Chu+qoVeq4Db7cbtQRFnL9jx1gdjmI5+CnREyDlM4XK5cPz4cVRVVcHpdCInJwclJSXIyMiY/80h4EKhyaPekOe2WOaL17YAfnMmtxJyYFiMyGMxG5Zq1YZmBxqa2W2H1NI5iZZOeYYiwiHng7mysjLU1dXh9OnTSEtLw6FDh7Bv3z6cO3cOCon7gXGxHJogCII1brcr4iNazpw5g6KiImRmZkKv16O4uBg9PT1oamqSHC8pNAmCWJKEsgEG4++vfg2jo4EzVkwmE0wm/+Etu92OdevW4ezZszCbzb7zW7duRX5+PgoLCyXFu6RW4BEEQXjxqksj4cSJE0EXuR04cAAHDx70O+dweIZw5hZpo9HouyYFKsYEQdz37N69G9u3bw84P7fgAoDBYADg6SHPxm63+65JgYoxQRD3PcGGI0JhNBqRkZGB9vZ2ZGdnA/AU4r6+PqxZs0ZyDPQAjyAIIkoKCgpw6tQp9PT0YHx8HEeOHMGqVatgsVgkt0k9Y4IgiCgpKiqC3W7Hrl274HQ6YbFYUFpaKnlaG0CzKQiCILiAhikIgiA4gIoxQRAEB1AxJgiC4AAqxgRBEBxAxZggCIIDqBgTBEFwABVjgiAIDqBiTBAEwQFUjAmCIDjgf41MB8VC1C9QAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vTNSmc0dLnsY"
      },
      "source": [
        "## **Model 4** : LSTM (fixed-length inputs + word2vec word-embeddings)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xCeCay0fLw2O"
      },
      "source": [
        "# Same two model as before but using the word2vec embedding\n",
        "class LSTM_w2vec_fixed_size(torch.nn.Module) :\n",
        "    def __init__(self, vocab_size, embedding_dim, hidden_dim, vec_weights, freeze_emd=True) :\n",
        "        super().__init__()\n",
        "        self.embeddings = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)\n",
        "        self.embeddings.weight.data.copy_(torch.from_numpy(vec_weights))\n",
        "        if freeze_emd:\n",
        "          self.embeddings.weight.requires_grad = False ## freeze embeddings\n",
        "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first=True)\n",
        "        self.linear = nn.Linear(hidden_dim, len(tag_to_ix))\n",
        "        self.dropout = nn.Dropout(0.2)\n",
        "        \n",
        "    def forward(self, x, l):\n",
        "        x = self.embeddings(x)\n",
        "        x = self.dropout(x)\n",
        "        lstm_out, (ht, ct) = self.lstm(x)\n",
        "        return self.linear(ht[-1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "M8-13iXKLnsd",
        "outputId": "8ca5405a-7238-4761-ca3f-7d4d385fba4e"
      },
      "source": [
        "w2vec_fixeds_notfreeze = LSTM_w2vec_fixed_size(vocab_size,\n",
        "                         embedding_dim = w2vecembedding.vector_size,\n",
        "                         hidden_dim = 50,\n",
        "                         vec_weights = pretrained_weights,\n",
        "                         freeze_emd = False)\n",
        "train_model(w2vec_fixeds_notfreeze, epochs=300, lr=0.01)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train loss 2.564, val loss 2.526, val accuracy 0.147\n",
            "train loss 1.591, val loss 2.598, val accuracy 0.176\n",
            "train loss 1.416, val loss 2.925, val accuracy 0.152\n",
            "train loss 1.354, val loss 3.103, val accuracy 0.163\n",
            "train loss 1.243, val loss 3.046, val accuracy 0.189\n",
            "train loss 1.099, val loss 3.115, val accuracy 0.219\n",
            "train loss 1.076, val loss 3.349, val accuracy 0.195\n",
            "train loss 1.072, val loss 3.422, val accuracy 0.189\n",
            "train loss 1.174, val loss 3.101, val accuracy 0.168\n",
            "train loss 1.116, val loss 3.091, val accuracy 0.235\n",
            "train loss 1.062, val loss 3.214, val accuracy 0.237\n",
            "train loss 1.041, val loss 3.386, val accuracy 0.227\n",
            "train loss 1.036, val loss 3.439, val accuracy 0.224\n",
            "train loss 1.030, val loss 3.522, val accuracy 0.213\n",
            "train loss 1.033, val loss 3.610, val accuracy 0.224\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 664
        },
        "id": "1aJBz9YGOWcr",
        "outputId": "d3e4bf5d-8c6e-43e4-8a1d-680e6adaeb89"
      },
      "source": [
        "print(\"***Model 4 : LSTM (fixed-length inputs + word2vec word-embeddings)***\")\n",
        "pred_vs_true_m4 = matrix_pred(w2vec_fixeds_notfreeze, val_dl)\n",
        "diagnostics(w2vec_fixeds_notfreeze, val_dl)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "***Model 4 : LSTM (fixed-length inputs + word2vec word-embeddings)***\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           A       0.00      0.00      0.00        22\n",
            "           B       0.00      0.00      0.00        12\n",
            "           C       0.00      0.00      0.00        13\n",
            "           D       0.00      0.00      0.00        22\n",
            "           E       0.17      0.15      0.16        20\n",
            "           F       0.16      0.14      0.15        29\n",
            "           G       0.09      0.05      0.07        19\n",
            "           H       0.58      0.43      0.50        65\n",
            "           I       0.50      0.11      0.18        18\n",
            "           J       0.15      0.10      0.12        21\n",
            "           K       0.17      0.54      0.25        61\n",
            "           L       0.00      0.00      0.00        17\n",
            "           M       0.11      0.05      0.07        37\n",
            "           N       0.00      0.00      0.00        19\n",
            "\n",
            "    accuracy                           0.20       375\n",
            "   macro avg       0.14      0.11      0.11       375\n",
            "weighted avg       0.20      0.20      0.17       375\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEDCAYAAADgNmW7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd1gU1/rHv1vpS0dhRUVFEEukeEVjj71rYg8YE03uL2qa5uaa5N6YrldjmiYSNRFL1GjsxKjYJWLBjqCoWFCUztJ2ly2/PzasrFuY3T3LHGA+zzPPw87OvLxz5uzZs2fOfIan1Wq14ODg4OBgFT7bCXBwcHBwcI0xBwcHBxVwjTEHBwcHBXCNMQcHBwcFcI0xBwcHBwVwjTEHBwcHBVDVGCcnJyMsLAy3bt2yK06HDh0wZswYjB49GuPGjcP58+ftipefn4+3334bAwcOxPjx4zFr1ixkZ2fbnNeIESMwevRo/Pzzz9BoNDbnVROvZvnpp5+IxcrJybE5VkFBAebNm4fnnnsO48ePx6RJk3Dw4EGr40RGRhq83r59Oz755BOb8zIXl+14tfc/duwYhgwZggcPHrCWTw1hYWGYP3++/rVKpUJsbCxee+01m2ItWrRI/3rNmjX4/vvvieTZWBCynUBt9u7di+joaCQlJeGNN96wOY6zszN27doFADhx4gSWLVuGDRs22BRLq9Vizpw5GDt2LL7++msAQGZmJgoLCxESEmJzXoWFhZg3bx7Ky8ttPtba8eyFVCytVovZs2dj7Nix+OqrrwAADx48wOHDh+2O3dg5deoUPvvsM6xZswZSqZTtdODq6oqsrCzI5XI4OzsjJSUFzZo1symWWCzGgQMH8Oqrr8LHx4dwpo0DanrGFRUVSEtLw+eff46kpCRiccvLyyGRSGzePzU1FUKhEFOmTNGvCw8PR0xMjF15+fr64tNPP8XGjRvRmO67SU1NhUgkMigvqVSKuLg4FrOin7Nnz+LDDz/EypUr0bJlS7bT0dO3b18cPXoUAJCUlIQRI0bYFEcoFGLSpElITEwkmF3jgpqe8aFDh9C7d2+EhITA29sbV69eRadOnWyKJZfLMWbMGCgUCuTn59tVAbKystCxY0eb97dEcHAw1Go1CgsL4efnZ/X+NcdZw2uvvYbhw4fblEvtWC1atMCKFStsipOVlYWIiAib9rWUEwCUlpZiwIABRGLThFKpxOzZs7Fu3Tq0bduW7XQMGD58OH744Qf0798f169fx/PPP4+0tDSbYk2bNg2jR4/GzJkzCWfZOKCmMU5KSkJ8fDwAXQVISkqyuTGu/ZP7woULeO+997B3717weDxi+dIAjcMUT/Pxxx8jLS0NIpEIv//+u105bd++HVevXiWdIusIhUJERkZi27Zt+PDDD9lOx4Dw8HDk5ORg79696Nu3r12x3N3dMWbMGKxbtw7Ozs6EMmw8UDFMUVJSgtTUVHz44YcYMGAA1qxZg3379hH5+R4ZGYni4mIUFRXZtH9oaCjS09PtzsMU9+/fh0AggK+vr0Pis0FoaCiuXbumf/3RRx9h7dq1KC4uZjEruuHz+fjmm29w5coVrFy5ku10jBgwYAD+97//2TxEUZvp06fj999/R1VVFYHMGhdUNMb79+/HmDFjcOTIERw+fBjHjh1DixYtcO7cObtj37p1C2q1Gl5eXjbtHxsbC6VSiS1btujXZWZm2p1bUVERPvroI0ybNq1R9dhjY2OhUCjw66+/6tfJ5XIWM2oYuLi4ICEhAXv27MHWrVvZTseAF154AbNnz0ZYWJjdsby8vDB06FBs27aNQGaNCyqGKfbu3YtZs2YZrBs8eDD27t2Lbt26WR2v9lijVqvF4sWLIRAIbMqNx+Nh+fLl+OKLL7Bq1So4OTlBKpXi/ffftzkvlUoFgUCAMWPGYMaMGTblVTteDb179zaYisQGPB4PK1aswJdffonVq1fDx8cHLi4urOflKFQqFcRiMZFYXl5eWL16NaZNmwYfHx8899xzNsWpqqpCnz599K9nzJhhVz1r3ry5fgiRBC+//DI2btxILF5jgccpNDk4bCczMxMffvgh19PjsBsqesYcHA2RTZs2Yf369Tb9SuLgeBquZ8zBwcFBAVRcwOPg4OBo6nCNMQcHBwcFcI0xBwcHBwVwjTEHBwcHBXCNMQcHBwcFcI0xBwcHBwVwjTEHBwcHBXCNMQcHBwcFcHfgcXDUAwkJCUhPT0d6ejpycnIglUrNPv0kJSUF+/fvR3p6Om7cuAGlUol169ahe/fu9Zx1wyZJxFxsNKL6ugMzYUa9Nsa9Rh0zuf7NWW0xYXQLHPsrH6lpRWgV7IYXRgbhUnop3vrPZTx9j+DJPX3NxqMh1qT5dw3Wd2rnjHEDPZF1V4GiUhXUaqBloAj9urmjUq7Be1/nQlZu+Cy8LUtbAQBGvZZhsH76OH8M7+eNM5fKcT27Ciq1Fl3C3NA7RoLs+3LMX3wHymrjmyr3JHQgcow1x0nyXNIaCyBT/nsSOmDZsmVwcfNEUOsIuBSXoUKhxdZU088//P3n3bh8KgkBLdrBt3lb5N7LwNFrWtzTajAhVvdjluS5JFFfAV2dHTjF2GaYvCkGsjIVsu5UIjTEFZVVarz4xhWTx157n6YG6z3jkJaueH6kFEf/yseHXz7x4OY+rsLbr4ViYJ8AHDyW16BjXb0px9WbxhrJjNsKvB3vj34x7th9VMYoVsr5MmzdV4hK+ZMPw5/HS/AwT4lJw/0w6FkvJB1l5g4meYyk49Eay9byT05ORuoD3XPtvnt/FJSKSrP/Y9Dzb2HMSx9DKBLj5B8/I/dehtltHXGMJOsrAMS9eRm5eUoAwKr/dYSLc/2MjvJEDUtNy/qY8cA+AeDzefhtl+HTcPfsz0WVXI3B/QIafCxz5BerAABuLsxPw827coOGoIYTZ3UfjlZBToxjkT5GWsufZCxbyz84OJjx/5D4NINQZJ2Wk9b6CkDfENc3AhcB44UGWO8Zh4d6QK3WIuOG4TetslqLrNvlCA/1aPCxahAJAWcxHyIRDy2aiTB1hDcA4EKm/U898PPWncqSMhXjfUgfI63l74hz+TS2lD9JGlp9rQ/4wobVM7a6MT537hySkpLw0UcfEUnAz0eMUlk1qlXGA1oFRQp0ifCEUMiDysT7DSVWDQO6e+DlcU8eU55XpML3GwuQma1gHMMUfB4waYQfVGotjp1h/vOR9DHSWv6OOJe1sbX8SdKQ6mt90dCGKRg1xteuXcOePXuwf/9+SKVSDB48mFgCzk4CVKtMX8hQKjV/b8NHuUrdYGPVcPZqJR7mVcNJzEOIVIzojq7wcLN/pGjmpGbo0NYViTvy8OAx85+EpI+R1vJ3xLmsja3lT5KGVF/ri0bTM87OzkZSUhL27t0Lb29vDB8+HFqtFuvXryeagFyhhrez6fExsZj/9zamK1lDiVVDUakaRaW6D8O59CqcvlKJL94MhJOYh52HbetRTRvtj1H9ffDn8WJs+7PQqn1JHyOt5e+Ic1mDPeVPkoZSX+uThtYzNvs1N2zYMKSmpiIhIQGbNm1CXFwc+Hzy34oFRUp4SkQQmfgW8/NxQnGpkvFPK1pjmeNebjXuPFBiUE/bxiynjPTD5BF+OJhSghUbH1m9P+ljpLX8HXUu7S1/kjSE+lrf8IU8xgsNmG1dly9fDn9/f8THx+PDDz/EqVOn4IiHgmRmlUEg4KFDe4nBerGIh9A27rh+s7zBx7KEWMSDu6v1X3JTRvph6ih/HPqrBN+vz7Xpf5M+RlrL3xHnkkT5k4T2+soGAjGf8UIDZrMYOHAgvv76a+zbtw/du3dHYmKi/vHyJ0+eJJbAoRP50Gi0mDhGarB+1JBAuDgLcODo4wYfy9PDdDF3bOuE4OYiZN217oLI5BG6huDwqRJ8uy7X5GR+JpA8RtLxaI0FkCt/ktBcX9mCx+cxXmigzgt4rq6uGDVqFEaNGoXS0lL8+eefWLVqFXr16kUkgdt3K7A96SFeGCXF5wsicCqtCK1buOKFUVJcuFJi1U0HtMaaOd4X3hIBrt6Uo6BYBZGQhzYtxOjZ1Q1VCi3W72F2kwYADO/njWmj/ZFXWI2LmZXo+w/DnlCJTI2LGRX1foyk49Eay9by37lzJ46c1s0BrigrhlpVjSO7fgQAePkFIfLZMfptH927jowLulul72VdAABc/GsX7t5IQ/55HuLi4hx6jCTrKwAM7OWDZv66+ddeEiGEAh6mjQsEADzOVyD5ZJFV8ZjCE9DR42WKVVPbPD09MWnSJEyaNIloEt+tvolHeXKMHhKIHt18USqrxra9D7FmY7bVvQ4aY6VcrECfaDf0jnaDxE0AQIv8YjWSU8uw+6gMhSXMr3C3b+UMAAjwFeGdGUFG71+5XsG4MQbIlhfpeDTGsrX8f//9d5w5c8Zg3aHt3wEAWod3M2iMH969pn+vhvPHt+v2ATB69GiHHiPJ+goAw/r745kIw3HmGRN1PfhL18oc1hjzBXT0eJnC+k0fAKDRAJt35mDzzpxGGSv1UiVSL5m//dUavknMxTeJ5MYoSZYX6Xg0xrK1/NevX2/WRfE0Ub3HIar3OJPv1bgpgFtG79FYXwFg3qfsSHhoGX5gChWNMQcHBwdpaLkwxxSe1hFTJDg4ODhYJq3/s4y3jT6S4sBMmNGwvjo4OBooCQkJeOONN/Dcc88hLCwMAwYMsLj9pUuX8NJLLyEyMhJRUVF45ZVXkJFRt72N4wmOmk3xww8/YODAgYiOjkb37t2Nzs21a9cwefJkPPPMM+jXrx/WrVvHLN/67Bk3FZ8xbbFq4jUFn7Epny5gvVO3xqdL6lyGhYXB2c0LzVtG4NHddDi5uOP1L0zL5R/cvoiNX8XBw6sZovu/CABIO7IBlWWF2LZ1M8LCwqitFzT5jC8O7s14264HTjDeNjs7Gz4+PvD09IRSqcSGDRvw888/4/jx46isrMTgwYMxdepUzJo1CxkZGXj11VfxySefYOjQoRbjsj5mTKu3lovVeHzGADmnrq15JScn49AtnUZz1ccjUW3BZ3xwy2cQCEV4cf5GeHg3AwB0iB6GnxYOw+LFi/Hzzz8Ty4vkMZqDNZ+xgy7ghYSEGLzm8/nIz89HWVkZDh06BD6fj9dffx18Ph9du3bFhAkT8Ouvv9bZGLM+TEGrt5aL1Xh8xgA5p66teTH1GRfl3UXunSsIjx6qb4gBwMO7GcKjh+Kvv/5Cfn4+sbxMQWvZWwuPz2e8yGQy5OTkGC0ymWkHx9GjRxETE4POnTtj0aJFmDFjBjw9PZGZmYmIiAgDdUSnTp2QmZlZZ76s94xp9dZysRqPz5gkjs4r947u57u0TaTRe9KQrric8jvS09MBGPb6mkLZW4tAxLyvmZiYiOXLlxutnzNnDubOnWu0vl+/fjh37hxKSkqwc+dOBAbqbmIpLy+Hh4dh+UgkEpSX1307OuuNMa3eWi5W4/EZk8TReZWX6n7+e3gZ9z7dvXQ95cePHwNo7rC8aC17a7FmmGL69OkYN854brdEIjGx9RO8vLwQHx+Pbt26oU2bNnB3d0dhoaG9TyaTwd3dvc4crG6Mi4qK4O3tDR6PzHgMrd5aLlbj8RmTxNF5VSt1T9EQCI11mDWPYqqqMn7SRlMoe2vhWWGZlEgkdTa85tBoNFCpVLh79y7Cw8Oxb98+aDQa/VBFeno6wsPD64xjMduLFy8iLi4Oc+bMwbVr1zBy5EiMGjUKPXv2xPHjx21K/GnkCjVEQtNp2OKt5WKxE4v23Ejh6LxEYhcAgFplPM6qqtatc3FxcWhetJa9tThqatu6dev04/ZFRUX4+OOPIRaL0bVrVwwePBhqtRo//vgjlEolLl++jK1bt2LKlCl1xrXYGH/yySd47bXXMGLECEyfPh2fffYZUlJSsGHDBixbtsyqAzAHrd5aLlbj8RmTxNF5uXvqhifKSoxnK5SX6MxrzZo1M3qvKZS9tTiqMU5NTcXYsWPRtWtXjB49GgUFBVi7di38/Pzg7u6O1atX4/jx44iJicHcuXMxe/ZsDBs2rM64FhtjtVqNXr16YdiwYfDz80PXrl0BAG3btrUqeUvQ6q3lYjUenzFJHJ1XYOvOAIAHty8Yvfcg+yJ4PB46duzo0LxoLXtr4QsFjBdr+OGHH5CSkoKLFy/i5MmT+PHHHw3OSUREBLZs2YLLly/j2LFjiI+PZ5avxTdrjbk4OzsbvEdqzJhWby0Xq/H4jEni6Lx8AlqheatOyEz7E2UlT2KVlTxGZtqfiI2Nhb+/v0PzorXsrYUv4DFeaMDiBbzMzExERUVBq9VCoVAgKioKAKDVaqFUkpk7SKu3lovVeHzGADmnrq157dy5EykpDwEAlWVF0KirkZL0AwBA4huEzrFj9dsOmvQBfl0Wjw1LphncgafVavHvf/+baF6OjgWw6DNuTNa2+roXnkZvLRercfmMSTp1bcnLlM/4+O5vAQAt2//DoDFu0TYKU99Zj+O7vsHx3d+CB0DaNgrjXvvW4lX5plD21mDNbAoaYH2eMUCnt5aLxX48krFIOnVtyWv9+vVYe5T5/2jRNhJT30l0eF71EYvzGTODisaYg4ODgzQNrTHmfMYcHByNkvuvP8942+AffndgJsyo156xOb3hlDHN0a61K9qHuCGwmRMe5Sts1hvyeMCE0VKMGRqE5gHOKClV4sjJfKzeeMfkRHVLqsrGEqsmnqnyt7bsAV35k4xlTqFpS5mRjAWQO5cJCQn442g6Ht1LR0lBDjx9pWYVmgDwIPsSju/8Gg+zLwE8HqRtItF//Hy8F9eBWF6WyoyGWPbCjRnbwCuTW+h9p25u1s35e5o3Zj7xsG7ecV/nYR0lRWgbd7Me1qYci2TZk4wF0FtmtsRatmyZ3mcsryyzGL+2z7j36DcB6GZTbFgyFWP/ofMZ03iM9RHLKghNv60vqGiM2XbNNtVYAFnXLMlYtJYZ5zNmt75aQ0MbM6aiH8+2a7apxgLIumZJxqK1zDifMbv11Rqs8RnTgMUs7t69i7S0NKP1aWlpuHfvnsOSshVana60xqIZWsuMbZ+xVqv922fsuLxojWUtjnJTOAqLjfEXX3xh0sPp7u6OL774wmFJ2UpdHlZvTzGEJuQnTTUWzdBaZo4uf+Y+Y8flRWssa3GUm8JRWGyMCwoKTF4sCAsLw4MHD0zswS5MPaxcLPqhtcwcXf6O9hkzgdZY1tKoesZlZeav+srlcuLJ2AutTldaY9EMrWXG+YzpKi+L8PnMFwqwmEWnTp3w22+/Ga3funWrSY0f29DqdKU1Fs3QWmacz5iu8rIEj8djvNCAxcb4/fffx/bt2xEXF4dFixZh0aJFePHFF7Ft2zZ88MEH9ZUjY2h1utIai2ZoLTPOZ0xXeVmiUc2m8PPzw+bNmzF79mxIpVJIpVLMnj0bW7ZsMelUZRtana60xqIZWsuM8xnTVV6W4AkFjBcaYHTTR2xsLGJjYx2WBNuu2aYaCyDrmiUZi9Yy43zG7NZXa6DlwhxTqLgDj23XbFOORbLsSXtraS0zzmfMXixr4PHoGH5gChWNMduu2aYci2TZk/bW0lpmnM+YvVhW0cB6xpxCk4ODo1FSsngO42293lvuwEyYQUXPmIOjsZOQkID09HSkp6cjJycHUqkUhw+bV2jWZsmSJVi9ejVcXV1x4YLxLAsO03BjxhYw55p9c9YTxV5qWpFOsTcyCJfSS00q9mpcp6acusmbYvQax9AQV1RWqW12I9uaF22xauJNmn/XYF2nds4YN9ATWXcVKCpVQa0GWgaK0K+bOyrlGrz3dS5k5cYT8rcsbUX0XJI4jzX7kK5jpM7lsmXL4OYhQes24SgukUFRrUFqZqnF4wGAu7dv4JdffoGzsyvUmieBSdYLkuU/6jXj52ZOH+eP4f28ceZSOa5nV0Gl1qJLmBt6x0iQfV+O+YvvQFltmNyehA4W/xcTeAI6ZkkwhfWeMa1KSFo1giRjXb0px9WbxndSZtxW4O14f/SLccfuozITezo+t8am40xOTsaDCt1c2/fnToZCbnxL89No1Gr8suJzdI7qCXlVBbJvmn9AMK2fIwBIOV+GrfsKUSl/8sX+5/ESPMxTYtJwPwx61gtJR4ttjm8WB80fXrJkCY4ePYrc3Fy4urqiT58+ePfdd+Ht7a3fJiwsDE5OThDU+kLYvNm8ixqgQKFJqxKSVo1gfSgJ84tVAAA3F+uqB8ncGpuOk6lCszYH9m7Bg/vZiHt1vsPyMgfJ8r95V27QENdw4qzui75VkBOx/1UbR92BJxAIsGTJEpw+fRq7du3Co0ePsGDBAqPtVq1ahQsXLugXSw0xQEHPmFYlJK0aQUeUl0gIOIv5EIl4aNFMhKkjdN/wFzLr7r05OjcS0F7+pijIy8X2XxMwdvJM+AUEUpMXSfy8dc1PSZnKMf/AQT3jd955R/+3r68v4uLiMG/ePLvjst4Y16XY6xLhCaGQV+++BZJ50RqrhgHdPfDyOB/967wiFb7fWIDMbAXjGI7KjQS0l78pElcuhn8zKYaOmcpoe1rL3hx8HjBphB9Uai2OnWE+FGYN1lzAk8lkkMmM85BIJJBIJCb2eMKpU6dMzv+eN28eqqurERQUhClTpmDixIkW4zBujIuKdBP2fXx86tjSOpgq9spVaqL/ty5I5kVrrBrOXq3Ew7xqOIl5CJGKEd3RFR5u1vcquHNJ5hhPHd+PK+dP4YMvV0EgYPYRpbXszTFzUjN0aOuKxB15ePCY3JCIAVbc9JGYmIjly42nt82ZMwdz5841u98ff/yBrVu3YsOGDQbr165di8jISPD5fKSmpmL+/PlQqVSYOtX8l6vFM63VarF8+XJs2KC7FVOr1UIgEODFF1/EnDnM5/BZQq5Qw9vZ2N0KsKuEJJkXrbFqKCpVo6hU9yE9l16F01cq8cWbgXAS87DzMPNeC3cu7T/G8rJS/Lr6a/QZOBqhHbow3o/WsjfFtNH+GNXfB38eL8a2Pwsd9n+smU0x/cXpGDdunNF6S73ipKQkLFy4ED/++KORvKlHjx76v/v06YOXXnoJu3fvtr0xXrt2Lc6fP49t27bpL0Dcv38fCxcuxNq1a/HSSy9Z2p0RBUVKtA52g0jIM/qJxaYSkmRetMYyx73catx5oMSgnh5WNcbcubT/GHduXg2Fogp9B4/F49z7+vVKhQJarRZ3796FWGzc6NJa9k8zZaQfJo/ww8GUEqzY+Mix/8yKYQomwxG12bp1K5YsWYKVK1ciOjq67lT4fNR1f53FfvyuXbvw1VdfGVwJDg4OxpIlS7Bz506GaVuGViUkrRrB+iovsYgHd1frhiq4c2n/MRbm50Ihr8In787Av/75vH65nZUOpUKOwYMHY9asWfWeFwmmjPTD1FH+OPRXCb5fn+vw/+cohea6deuwdOlSrFmzxmRDnJ6ejitXrkCpVEKlUiElJQW//PILRowYYTGuxSxUKpXJMWIfHx+oVGSugNKqhKRVI0gylqeH6dPfsa0TgpuLkHXXugt43Lm0/xhHjI/HnH99abQEBYdAJHbCt99+a3IaFa1lX8PkEbqG+PCpEny7LtehgiA9PB7zxQo+//xzlJeXIz4+HpGRkfrl4UOdle/x48f417/+he7duyM2NhaLFy/GW2+9hfj4eItxLQ5TiEQim96zBlqVkLRqBEnGmjneF94SAa7elKOgWAWRkIc2LcTo2dUNVQot1u+xbiI+ydwam45z586dOH35NgCgrLQEKlU1dv22BgDg5x+IZ/sPBwC0Czc9Tpz8x1YU5j/C0KFD/15jeAcerZ8jABjezxvTRvsjr7AaFzMr0fcfhr33EpkaFzMqrMqPEQ6a2nb9umUh1oABAzBgwACr41psjDMzMxEVFWW0XqvVQqkkdwWUViUkrRpBUrFSLlagT7Qbeke7QeImAKBFfrEayall2H1UhsIS66+8k8qtsek4TSk0t29MAACEd4rSN8b2QOvnqH0rZwBAgK8I78wIMnr/yvUKhzTGjep26IwM87dfkoRWJSStGkFSsVIvVSL1UqXd+dSGVG6NTce5fv16Ri4Kcyz4fKVD8jIHyfL/JjEX3yQ6fozYCM5nzMHBwUEBDczaxvmMOTg4GiXyTYsZb+s85T0HZsIMKhSaPB4wYbQUY4YGoXmAM0pKlThyMh+rN94xOVHdkkJzypjmaNfaFe1D3BDYzAmP8hU2KzRtzYtkrKe1l4H+QrwwyAshUjG8JQIIBEBBiRoXMqqw56gMJWWmx3m3LG1FpLwAXZmRjGVKuxgUIEb/7hJ0jXBHoL8IIhEPj/KrkZImw65DRVAojfsQexI6mMzLltwcUS+Wbjd+r1pRgat/bcCtS0koL34AvlAMT7/W6PCPiQiNGmsksZk/nk8sr5rcSNYxknnZTQPrGVMxTPHGzCce1s077us8rKOkCG3jbtbDao5XJrfQe1jd3OwbwCeZF6lYvp5CeEkEOHO10sBBPDDWHT27upp1EJuDZHmRjDXoWU+9A/fYmVK9AzdubAB6RUtMOnDrKzdS51Kr0WDf2teQd/cCQqPGomPPF6FSVuHWpT9wbNv7KM67he7D6ja2kc6LdB0j+TmyCm7M2Dpo9bDS4MA1BWkHMUlvLc0OXBrrRd79S3h8Jw2dno1Hj5FP5g5HxE7B1mUjkHnmN8aNMa11jPTn2yoa2GwK1r86aPWw0uDAtQZbHcQkvbU0O3BprBdKhW46l6vEcB+BUAxnN28IxS6s5GUOW+pYfeRlFh6f+UIBrPeMafWw0u7AJeUgbmg43IFbByTPZUBwZ4idJbh8fA08vKUICO4ClVKOG+d3ouBBOnqN/YiVvGogUcdY/XxzY8bWQauHlXYHLikHcUOiPhy4dUHyXDq5eGJw/Aqc2P4fHPr1bf16kZMbBk77Fq07DmQlrxpI1DFWP9+U9HiZYjHbVatW6f/et2+fwXvLli0jkgBTD2t9QzIvRxzj2auV+CzhMZb8kodtB0pQUaWxyUHckKhx4G7cne84B24dkD6XIidXeDcLRZfeL2Pgi9+h9/hPIfFticNb3kVOVgpreQFk6hirn28HuSkchcVS+OOPP1T98U8AACAASURBVPR///TTTwbvnThxgkgCcoUaIqHpNNh24JLKyxHHWFSqxpUsOc6lV2HrgVL8sLkA00Z6Y+wA5hrAhkR9OXDrguS5LHp0A7t/nAppaE90H/4uQjoOQni3FzD6tY1wdffDie3/hUbD7JZ0WusYq59vPp/5QgEWs6h9P8jT94aQulekoEgJT4kIIqHxtxPbDlxSedXHMdZ2EDc26tWBWwckz+WVk4lQqxRo02mIwXqh2AXB4X1RXvIQ5cUPzOztuLzMYUsdY/XzzRcwXyjAYmNce8L505PPrX2iqjlo9bA2JAdu7XjWOohpp74duHVB8lxWyHRaS63WuGeo1eguUDLtGdNax1j9fDemnnGNtS0yMhLXr19HVFSU/vWNGzeIJECrh5VWBy5pBzHNsOLArQOS59I7oC0A4EbaDoP1iioZ7l47DCcXT0h8W9Z7XiTrGKuf7wY2Zsy6tY1WDysNDlxTkHYQk/TW0uzApbFedHo2Hlnnd+HM/mUoenQDzVpHQVFZisyzW1FZlo9nR/8HfIY/oWmtY6Q/31bRwGZTsD61DaDXw8q2A9cUpB3EJMuLZgcujfXCw1uKsbN/w/lDP+DhrVTcurwPQpETfAPDETv8XwjpNJh5MIJ5ka5jJD9HVkFJj5cpVDTGtHpY2XbgmoK0g5hkedHswKW1Xkh8W6LfxEUEsqK3jpEsL2vQNrDboTmFJgcHR6Ok6shGxtu69J/mwEyYQUXPmIOjsZOQkID09HSkp6cjJycHUqkUhw8fNrnt0qVLce7cOdy9exdlZWXw9fVFeHg4Xn75ZXTv3r2eM2/AcGPG5jHnM35z1hPFXmpakU6xNzIIl9JLTSr2LHmDbY1lyoObvClGr10MDXFFZZWakQOXtlg1+7z8cd0XSyYOcke/aGc4O/Fx6EwlNu4znnr080cBRPMy5TOePs5fr9C8nl2lV2j2jpEg+77cpEJzT0IH4nWM1LlctmwZ3Nw9EdwmHK4lMlQpNThyxbTj4WjKeTSXtkX/zv3h6iaBrKQAp0/8gfj4eCxevBhjx44lUvdrjvNpn3Gnds4YN9ATWXcVBgrNft3cUSnXmFVoblnaiui5tBctN2ZsHbSqKgF69ZIkY9WmZXMhBsW6YOvBckweYv3NI7QqNGmoF8nJybhZ4gcA+OTt5yGXmx+TnffJGqN1/YdPxX/mjMRPP/2EsWPHGr1Pq0ITIK9DZUwD6xmzni3Nqkpa9ZIkY9XA4wEvjfLA1ZtKpGXYNleZVoUmDfUiODjY6n1q4+ziCjcPL8hkphtBWhWaAHkdKmMa0zzj+oB2VWVTYXCsCwL9hFjxG3veBybYotBsqPWiXFYMjVYLWXE+TiRvx6Oc23j++edNbkurQtMSjtahOmo2xZIlS3D06FHk5ubC1dUVffr0wbvvvgtvb2/9NteuXcMnn3yCjIwMeHt74+WXX0Z8fLzFuKw3xrSrKpsCfl58jO3njt3HK1BYqoGvJ+s/mExiq0KzIdYLeVUl5r/cX/9aJHZG70HP48MPPzS5Pa0KTXPUiw7VQcMUAoEAS5YsQWhoKGQyGd59910sWLAAK1euBACUl5dj5syZmDp1KhITE5GRkYFXX30VAQEBGDp0qNm4Fhvjhw8fIijIeMI9SZgq9spVdU80JxmrKRE/wgP5xWocOEVubqkjqFFoJu7Is0qh2RDrhVjshDf/uxIatRqF+bk4c+IPyKsqIZfL4erqarS9I47x7NVKPMyrhpOYhxCpGNEdXYlpWm09l9agdVBj/M477+j/9vX1RVxcHObNm6dfd+DAAfD5fLz++uvg8/no2rUrJkyYgF9//dViY2wx29mzZ+v/njt3rj35m4V2VWVjJ7azEyLairE+qQxqiovGHoVmQ6wXfIEAHbrEomPks+gz+AW8s3AVigseYfr06aiurjbanlaFpinqTYdqxZixTCZDTk6O0WJujL42p06dQnh4uP51ZmYmIiIiwK8lIOrUqRMyMzMtxmGs0Lx//36dSdlCQ1NVNiaEAmDyEA9cyVKitFyDAG8BArwF8PXSjbW5OPER4C2AixO7FzjsVWg2hnrBFwjwj97DcePGDZw7ZzzdjlaF5tPUpw5Vy+MzXhITE/Hcc88ZLYmJiRb/xx9//IGtW7figw8+0K8rLy+Hh4dhGUkkEpSXWzbUWRymsKTQJEVmVhm6R/mgQ3sJLl8r1a+vUexdSi+1sLfjYjUFREIeJG58PNPeCc+0N76i3fMZZ/R8xhlbDpRh/yl2nq1HQqHZWOqFUqmbblZSUgLAcKiivo7RHk1rvetQrfAUT58+HePGjTNaL5GY/xWQlJSEhQsX4scff0THjh31693d3VFYaNjjl8lkcHd3t5iDxca4RqGp1WqhUCgQFRUFQNdj5vF4OH/+vMXgTDh0Ih9xE1pi4hipQSWyVVVJKlZTQFmtxYrfjD+kHm58xI/wwJUsBY5fkCPnMTsP/ySl0GxI9aKiXAYnJxcIRSKD9Qp5Ff46vBN8Ph9dunQBcNPgfZLH6OnBR2mZ8ZBGjUIz/ZbxHOS6YEOHas1NHxKJxGLD+zRbt27FkiVLsHLlSkRHRxu8Fx4ejn379kGj0eiHKtLT0w2GMkzRqBSatOo4aY2l1sDknGLdbAoP5BWrrZpzTKtCk4Z6sXPnTqRcuAcAKJMVQ62qxh/bdM+Y9PEPRGzfkQCArGvnsDHhM0TGPoeA5i3h5OyKwrwHOH08CcWFjzFnzhxIpVI83RjTqtAEyOtQGeOgC3jr1q3DihUrsGbNGnTu3Nno/cGDB2Pp0qX48ccfMWvWLGRmZmLr1q1YuHChxbisT20D6FRVAvTqJUnGIgnNCk2268Xvv/+OM2fOGKzbvXkFACA0IlrfGEtbhqJLTF/cSE/DmRP7oFTI4e7hiVZtO2Lqqx9g7ktDjGKTPkbSCk3S55IpWjhmaPXzzz+HUCg0mjeclJSEoKAguLu7Y/Xq1fj444+RkJAAb29vzJ49G8OGDbMYl4rGmEZVJUCvXpJkLFMUlmoYeSyehmaFJtv1Yv369WZdFLXxbx6MuP/7yJa0qFVokj6XTHHU1Lbr1+s+/xEREdiyZYtVcalojDk4ODiI08DcFJzPmIODo1FSdPkE4219uvR2YCbMoEKhyeMBE0ZLMWZoEJoHOKOkVIkjJ/OxeuMdkxPVLSk0uVimJ/af3NPXpA5yypjmaNfaFe1D3BDYzAmP8hWMtJckY5GsF6TrGInjTN4Ug4SEBBz96zLu3c5AQd4D+PgH4osf95ncfseGb5F1LQ35j+6jqrIcHp4+aNGqPQaOjsc/p/QBQLZePK29DAoQo393CbpGuCPQXwSRiIdH+dVISZNh16EiKJSm+297EjoQrRd2Q4kAiClUDFO8MfOJh3Xzjvs6D+soKULbuJv1sHKxyMR6ZXILvZvXzc0+sQrJWEDjKrPaPuPKyjKL296+cRnSVu0RGTvQwGf89cJZaO6y2KRCk+QxDnrWU+8fPnamVO8fjhsbgF7REpP+YUuQrhdMcdSYsaNgvTGm1WfcFGIB9HqWG1uZNSSfMWn/sKP823XhqNkUjoL1rw5afcZNIRZAr2e5sZVZQ/IZk/YPO8K/zQRrboemAdZ7xrT6jJtCLJrhyox9n/HTONo/TJzGNGacnJyMx48fY9o03ZNTJ0yYgKIi3YT2d99916IOjim0+oybQiyaaeplRoPPuDb14h8mjIZXf+PTJLDYGK9evRpff/21/rVSqcS2bdtQVVWFBQsWEGmMafUZN4VYNNPUy4wGn3Ft6sM/TBpahh+YYjHb6upqBAYG6l9HR0fD29sbQUFBqKoiY/Gi1WfcFGLRTFMvMxp8xjXUm3+YMFrwGC80YLExfvpiwX//+1/93zXDFfZCq8+4KcSiGa7MDGHLZ1yf/mHSNLQLeBaz6NKlC3777Tej9Zs3b/5b42c/mVllEAh46NDe0ORU42G9ftOykJmLZXssmuHKzBhDn7EhjjjGevcPE0bL4zFeaMBiY/z+++9j+/btiIuLw6JFi7Bo0SLExcVhx44deP/994kkcOhEPjQaLSaOkRqst9VnzMWix81rD021zCrKZVCZGIYw9hkbQvoY2fAPk0bDEzBeaMDiBTxfX19s3rwZp06dws2bOn9q37590aNHD2IJ0OozbgqxADo9y0DjK7OG5DMm7R8mWS+sgZaxYKYwmmfco0cPog3w09DqM24KsWj2LDemMmtIPmPS/mG2/Nu0jAUzhfWbPgB6fcZNIRbNnuXGVGYNyWdM2j/saP+2ORpaz5hTaHJwcDRKsm/drHujvwlp286BmTCDip4xB0djJzs7G7t370ZKSgru3bsHhUKBli1bYujQoZg+fbrRjRy3b9/G0qVLcfbsWVRXVyMiIgJz58516HBhY4PrGVvgaW8qAEwf56/X9V3PrtLr+nrHSJB9X25S17cnoYPJePbEMuWHfXPWEyVhalqRTkk4MgiX0ktNKglrHLiT5t81WN+pnTPGDfRE1l0FikpVUKuBloEi9Ovmjkq5Bu99nQtZueGE/C1LWwEw7dNN3hSjVxKGhriiskrNyA/L5FFKEwe5o1+0M5yd+Dh0phIb9xlPifr5owCz3mBbykwjr4L8yjko7mQBajWc2neEa1RPKB/cQd5XHwB/zy5w6vAM/P65AKqCR6hMPQJ1eRlEgcFw6/kcoFKB7+ZONC+A3LlcunQpNq5fhz6RHdGpXSsIBQKkZdzEwdMXERociF8WvglnsRgAkPO4ANMXfgMBn48pQ/rA3dUFO46m4lZOLlavXoOePXsSrRck6n5NmZHMy15u3spmvG27tiF2/z97Yb1nTFLXRzIWSSXh1ZtyXL1p/HjzjNsKvB3vj34x7th9lPn9/o5SErZsLsSgWBdsPViOyUOsF8vYWma5//kntPIn46kVKQehysuFZOjzcOsxABXH9wMAPPqPADQa5C/7DzQVT5zA1bn34TP1n8TzMoWt53LIkCGYFtMa7q4u+nUvPNcTwc388PPuZOw6dhqTBumeNrH8tySUVVRh/afvIKyV7kLXiF4xmPjv/+Hjjz/Gn3/+aTI3UvWCBuUoCbTsSymtgvVsSer6SMYirV00RX6xzn7l5mLdaXCEkpDHA14a5YGrN5VIy1DYFMPWMqvdENdQef4vAIAosOWTHJ1doa1WQlNp2FvXlFr+gqXhXHbu3NmgIa5hcGxXAMCtHN3dbVVyBY5fSEd0h3b6hhgAXJ2dMLZfd9y5cwdXrpjuVZKqFzQoR0nQqG6HZhOSuj5bYjlCSSgSAh6ufPh4CtClvTNmveALALiQScbzYQ+DY10Q6CfEhj8sP4XCEiTLTOitKxtN2ZO7zRQZl8B3cYV33ByIpK3A9/SGU/gz8BwXh+pc8zMIaD6Xj4tKAQC+El0OWfdzoaxWoXO7VkbbdmqrW2euMSZFQ1WOPk1Da4xZH6YwBUldn62xHKEkHNDdAy+P89G/zitS4fuNBcjMtq0nSgo/Lz7G9nPH7uMVKCzVwNfTtu9oYmXG48FjyPPQqlWoPHdSv1p2cAf4HhK4xfaHW7cnD5CsSj+PorXfQrok0bF51YLEuVRrNFiz8yAEAj6G9IwCAOSX6BrnAG9Po+0DfHTrHj9+DCCM8f+xloaoHDUFLY0sU6hsjEnq+myN5Qgl4dmrlXiYVw0nMQ8hUjGiO7rCw439HyfxIzyQX6zGgVPmHwXEBFJl5vX8S3BqE4bS3b9ClVdrvqtGA3VJEeTXr6Dq0hloKsvh1CYc7n2GwuelNx2eV21InMuvNuzE5Zt3MHvCcLQO1P30lyt0FytFIuOPplgkAgBixkRzNETlqCkc1RgnJSVh48aNyMzMREVFBa5fN5xHHRYWBicnJwgET26z3rx5M8LCLH+BWmyMP/30U/AsSDTMia7tgaSuz55YcoUa3s5ik+/ZqiQsKlWjqFRXgc+lV+H0lUp88WYgnMQ87DzMjrA7trMTItqKsfiXEqjttEiSKDPJiElw7zsM5ScPouzgToP3vF+cDaeQ9nj0xTv6GRbyy2ehyn8E78mzHJrX09h7Ln/ctg+/HTyJcf1jMWP0QP16Zyddg1tdbTykpvz7mF1cjMeeSeKI8mIDjdYxHR2JRIKpU6dCLpfjgw8+MLnNqlWr0L17d6viWsy2U6dO6NixIzp27IjDhw/r/65ZSENS12dvrPrQLt7LrcadB0oM6snOGJxQAEwe4oErWUqUlmsQ4C1AgLcAvl66b3QXJz4CvAVwcWLWw7C3zCTDJkAy9HlUnDqCki2rDN4TePvCrVtvyNPP6xviGqounHJoXkyw5lwmbP8Ta3YdxKg+/8D7MyYYvOfvpRuKyCsuNdov7+/x5WbNmtmVa100BuUo4Lgx4969e2PkyJF2P9fwaSz2jMeNG6f/OzEx0eA1aUjq+kjEyswqQ/coH3RoL8Hla08+GDVKwkvpxh8WWxCLeHB3ZWeoQiTkQeLGxzPtnfBMe+OZJj2fcUbPZ5yx5UAZ9p+q+6exPWUmGTYBkuETUHH6KIo3rTR6X+D59/gs30RZCSxbt2g6lwnb/8SqHQcwslc3/OeViUa/PNsFB0IsEuLKzbtG+169pVvXqVMnYK+x2Y0U9VVejsaaRlYmk5l82KtEIoFEIjGxh2XmzZuH6upqBAUFYcqUKZg4cWKd+zAeM7Y0XGEvJHV9pGIdOpGPuAktMXGM1KBC2qIk9PTgo7TM+Gddx7ZOCG4uQvot43mr9YGyWosVvxl/sDzc+Igf4YErWQocvyBHzmNms1BsLTOPoc/rGuIzx1C88UeYOmnVeQ+hVavh0qUbSvdsgrbqyfi2W/d+DsnLFPacy1U79mPVjgMY/mwM/jtrEvgmvlhcnZ3Qu2tHHDl3GTfuPkD7v6e3VcoV2Hn0NFq3bv23QjONcc7WQrK82ESrZd5mJSYmYvny5Ubr58yZg7lz51r1f9euXYvIyEjw+XykpqZi/vz5UKlUmDp1qsX9WL+AR1LXRzIWSSXhzPG+8JYIcPWmHAXFKoiEPLRpIUbPrm6oUmixfg+zG1FqIKUkVGtgck6xbjaFB/KK1VbNOba1zDxHTIKqKB+K61fgGt3LMMeyEiiuX4G2sgLlR/+Ax3Oj0Oy9/6Hir0PQVJRD3CYMrjG9oMp/BKF/c6J5mcLWc7lx40YkbN+P5r7e+EenUPx56rzB+z4SD8R21l3gmTNpOM5eu4E5/0vA1KF94ebijB1HU5FfXIqExUvMdoxI1QsalKMk0FjRM54+fbrJX/629Ipr37Lep08fvPTSS9i9e7d9jXFkZKT+xMvlckRF6abfaLVa8Hg8nD9/3tLujCCp6yOt/iOlJEy5WIE+0W7oHe0GiZsAgBb5xWokp5Zh91EZCkusuyrNlpKQCbaWmdDHHz5xc4zWK7LSkX9dN6+2dOd6qPIewq3HAHgMHgeeUAh1SREqTh6EbN9vCPpyDfG8nsbWc1kzN/hRYTEWJmwyej8qvK2+MQ5u5o81/3kD3/+2F2v3HkK1So3w1i3w3buvonfv3kb71kCyXrCtHCWBNRfwbB2OYAKfzwcT64TFxvjChQvEEjIHSV0fafUfKSVh6qVKpF6yb9pYbRytJCws1TDyWJjCljLLmVv3eFoNFX8dQsVfh+olL1PYei4XLVqED8abb0ifJkTaDMvefsWq/0GyXrCtHCWBo6a2qdVqqFQq/YNhFQrdr0eRSISMjAxoNBqEhYWBz+fj9OnT+OWXXzB79uw647I+TMHBwcHhCKwZM7aGXbt2YcGCBfrXNY/BWrduHSoqKrBkyRI8evQIAoEAQUFBeOuttzBlypQ643I+Yw4OjkbJ2evGD241R7cwLwdmwox67Rmb0xvyeMCE0VKMGRqE5gHOKClV4sjJfKzeeMfk5PIavaGpeFws05PxzekNp4xpjnatXdE+xA2BzZzwKF9hs3bR1uN8WlMJAIH+QrwwyAshUjG8JQIIBEBBiRoXMqqw56gMJWXGY7NblrYyqWkNChCjf3cJuka4I9BfBJGIh0f51UhJk2HXoSIolMb9kRq1Kokyq9FBrthn/F5lWQFS932HO9eOobKsEK4efmjbZSBih74BJ1fjMczZw0Asr5rcHF0vbK2v9uKonrGjoGKY4o2ZT9ypm3fc17lTR0kR2sbdrDuVi0Um1iuTW+hds25u9j8ll1Ruvp5CeEkEOHO10sAbPDDWHT27upr0Bptj0LOees/1sTOles913NgA9IqWmPRcW4JUmVWWFWLLsomokOWhU89J8G0eisJHWbiSshkPbp3DhDc3QSRmfrcdyXNJMhbJ+moN9N8jaAjrjTFJdyoXi13XLA3eYFOQ9FwD5Mrs3MGVKCt+gCFxXyEseqR+fWDrSOxfPw8Xjv6Cfwx+vd7zIhmLdH21BkfdDu0oWM+WpDuVi8Wua5YGb7ApSHquAXJllnPzNIQiZ7SPGmGwvn3kcAhETrh2ejsreZGMVR91whxaLY/xQgOs94xJulO5WOy6Zh3lDXYW8yES8dCimQhTR3gDIOOAJunMtgW1SgmByMnoJg4enw+hyBmywvuoKi+Ci7uPmQj0w2Z9bWgKTdZ7xnW5U709xRCaEJZwseyPRRpH5DaguwdWfxKMH//TAh+82gxuLnwiDmiSzmxb8WkeCkVlKfJzDC865udkQFGpuw25rJjcvHk2YLO+arTMFxpgvWdM0p3KxWLXNUurN9gUJJ3ZttK173TcvpKMfYlvoc+49+EbGIrC3Cwc3/EF+AIRNOpqqKrZfwqMPbBZXxtaz5jx7dC1IXk7NEl3KheLXdcsjd5gU5B0ZtuDtG0Mhk5fhmPbP8fun14FAPD4AnSMfQFV5UW4dfkgxM7urOVHAjbrKy1jwUxh/XbogiIlWge7QSTkGf2UsdadysVi1zVbH7nV9gbb0hiTdGaTILTrMLTtMhiFD29AqaiAd0AIXD18sWXZC+DzhfD0M34WXkOCzfqqbmCNMetjxplZZRAIeOjQ3nCCe4079frNcjN7crHsjUWa+srNVgc0SWc2Sfh8AfxbdIC0bQxcPXxRIctHfk4GpO26WTXPmEbYrK8NbTYF643xoRP50Gi0mDhGarDeFncqF4td1yzJ3Dw9TFfNGm9w1l3rLuCRdGY7Eq1Gg2PbP4NGq0bMoH+ynY7dsFlftVrmCw2wfgGPpDuVi8Wua5YGb7ApSHquAXJlplRUYMuyCWjbZRAkPi2glJfhxvm9yLufjh4j3kZwaCzjnEjmRTIW6fpqDY3qAl59QdKdysWyLhZp1yzb3mBTkPZckyozgUAE/6Bw3EjbgwpZPoRiFzQL7owxr61Gqw7MdZuk8yIdi2R9tQZapqwxhYrGmKQ7lYtlHaRds2x7g01B2nNNqswEQjGGTl9GJBZA9lzS6ka27v82rJ4xp9Dk4OBolOw9z/zOypFR7PdL2c+Ag6MJkJCQgPT0dKSnpyMnJwdSqRSHDx9mtO+SJUuwevVquLq61st008ZCQ+tmUuEzfnPWE8VealqRTrE3MgiX0ktNKvYsuX4bW6yn/bzTx/nrdZDXs6v0OsjeMRJk35eb1UHuSehg0lubvClGr0oMDXFFZZXaLp+xLcdJa14AmfLfk9ABy5Ytg5uHBK3bhKO4RAZFtQapmXU/8v7u7Rv45Zdf4OzsCnWtQVASdazmOB1d/rbmZS+0TFljCus9Y1r1krTGolUHCZA9TlrzsrX8k5OT8aBCN4vj/bmToZDXfZuzRq3GLys+R+eonpBXVSD7prE43xHHCDQWhaZDwjoM1ucZ06qXpDUWrTpIgOxx0pqXreUfHBxsRcY6Duzdggf3sxH36vw6t6VVrcquQpObZ2wVtOolaY1lDrZ1kAC9es+GWP4FebnY/msCxk6eCb+AwDq3b8plb46Gdju02cbYnCQIAMRiMVq2bIm3334bPXr0sCuBuhR7XSI8IRTyGN2/3hRimYIGHSTg+OOkNS9HlH/iysXwbybF0DFTGW3fVMveErT0eJlitjG2dNVWrVYjKysL8+fPx969e+1KgFa9JK2xTEGDDhKgV+/Z0Mr/1PH9uHL+FD74chUEAmY/Xptq2VuioTXGNo0ZCwQChIeH48UXX7Q7AblCDZHQdBq26CUbe6ynoUUHCTj2OO2hIZV/eVkpfl39NfoMHI3QDl0Y79cUy74uNFoe44UG7LqAN3nyZLsTKChSwlMigsiE7d8WvWRjj1Ub2nSQjjpOWvNyRPnv3LwaCkUV+g4ei8e59/WLUqGAVqvF3bt3kZtrfDdhUyt7JjjqAl5SUhKmTp2KqKgohIWFGb1/7do1TJ48Gc888wz69euHdevWMYrL+mwKWvWStMaqgUYdJK16z4ZU/oX5uVDIq/DJuzPwr38+r19uZ6VDqZBj8ODBmDVrltF+TansmaLWMF+sQSKRYOrUqXj//feN3isvL8fMmTPRq1cvnDlzBt988w2WL1+OP//8s864rDfGtOolaY0F0KuDpFXv2ZDKf8T4eMz515dGS1BwCERiJ3z77bdYsGCB0X5NpeytwVE+4969e2PkyJEmpyseOHAAfD4fr7/+OpycnNC1a1dMmDABv/76a51xWZ/aRqtektZYtOogAbLHSWtetpb/zp07cfrybQBAWWkJVKpq7PptDQDAzz8Qz/YfDgBoF256nDj5j60ozH+EoUOH/r3G8E43WtWqrCo0rfiSlMlkkMmMZ8JIJBJIJBITe5gmMzMTERER4POf9HM7deqErVu31rkv640xQK9eksZYtOogayB1nLTmZWv5//777zhz5ozBuu0bEwAA4Z2i9I2xPdCqVm0ICs3ExEQsX77caP2cOXMwd+5cxnHKy8vh4WFYbhKJBOXldQ/HUNEY06qXpDEWrTrIGkgdJ6152Vr+69evZ+SiMMeCz1fWuQ2talW2FJrWNPTTp0/HuHHjjNZb0ysGAHd3dxQWGs6qkclkcHev+8GyT5OF2wAAG7NJREFUVDTGHBwcHKSxpjG2djjCHOHh4di3bx80Go1+qCI9PR3h4eF17sv5jDk4OBolq5KZbztrIPNt1Wo1VCoVzp49i1deeQWXL18GAIhEIlRWVmLw4MGYNm0aZs2ahczMTMyaNQsLFy7EsGHDLMat18Z40vy7RusC/YV4YZAXQqRieEsEEAiAghI1LmRUYc9RGUrKjO/M2bJU9/jyp/WGQQFi9O8uQdcIdwT6iyAS8fAovxopaTLsOlQEhdK0WhIwrSTk8YAJo6UYMzQIzQOcUVKqxJGT+Vi98Y7Jieo12j9TSsIpY5qjXWtXtA9xQ2AzJzzKV1hUEiZvigEAzF5aYvTeivleJvdRKLV45zvTP4VXzPfCyx/XfbFELAQ+fd0X/t4CHDpTiY37jMe6fv4owGRetuS2Yr4X0VjmFJo0nMvyjFtwCgoAXyRE1b1c5P15DLe/WgPFo3z9tiFvzUCzkf3h1j4EIh8vVBeVoOJ6NrKXr8PjXckYUa0bPiBRX2uOk8TnCNB9lkjmZS8JB5hv+9pg5ttu377d5IyWdevWoXv37rh27Ro+/vhjZGRkwNvbG6+88gri4+PrjMv6MIWvpxBeEgHOXK1EUakKajXQMlCEgbHu6NnVFe99nQtZObOJgIOe9dS7Zo+dKdW7ZuPGBqBXtMSs69ccb8x84mHdvOO+zsM6SorQNu5mPazmeGVyC70f1s1NwHxHM9y8r8LJy4ZPSLZ2vqQpxvZ3h4erfXckkcyNVCwazuWjXQchf/AYGpUKkk7t0XLmRARNHIETMWOgzNddEPPq1gWVdx8gb99xKAuLIfb2RPMXhiJm2wpc/+jbejlGmj9H1uCouOPHj8f48ePNvh8REYEtW7ZYHZf1xvjqTTmu3pQbrc+4rcDb8f7oF+OO3UeZyVdIun5p9cPWUFCqwdmMartiPE3L5kIMinXB1oPlmDzEdpsWydxIxKLlXF7/z9cGrwtPnEP05m/RIn48bn+1GgBwYdrbRvtlf5eIXme2o+38mWZj0+BsdnRe1tLQBmBZv+nDHPnFOhWhmwvzFEm6fmn1w9ZGwAecRGRi8XjAS6M8cPWmEmkZirp3qAOSudkbi9ZzWXVPl4/I2/KFI61aDfmDxxC4uZjdhgZns6PzshaNlvlCA6z3jGsQCQFnMR8iEQ8tmokwdYQ3AOBCZt1PRKgLW1yztPpha4hsL0K3CE8I+DyUVWqQllmNPSerILexnRgc64JAPyFW/Ga/7IZkbiRi0XIu+U5iCNzdIHAWw71DO4R/qZPG5+0zHmcVeXuCJ+BD7OeN5s8Pg/+Q3ig8ehp+A0wra2l1NrNZ9tZdDmNfFmSxMc7NzUVgoGmx9ZEjR9C/f39iiQzo7oGXx/noX+cVqfD9xgJkZtvXS7PVNUurHxYA7uSqcP56NfJLNHAWAx3biNAvygmhwUJ89WsZFFb+qvfz4mNsP3fsPl6BwlINfD1t/8FEMjdSsWg5l8GvTECnb/+rf12ZnYML8fNRnJJmtG2/a/sh9tN1SDTV1Xi0/QCuzl2IQbmpJmPT6mxms+zV9WsLtRuLjfGMGTOwevVqtGjRwmD9tm3bsHLlSqKN8dmrlXiYVw0nMQ8hUjGiO7rCw83+URRbXbO0+mEBYMlGwxkOZ65V42G+GqN7u6BflBP2n7buCyx+hAfyi9U4cKqSqtxIxaLlXD7elYzyzNsQurtC0jUCzUYN0De4T5M2YQ74zk5wljZD4PNDwXdxgsDDzWxsWp3NnM+YORZbuwULFuCVV17BnTt39OsSEhKQmJiIDRs2EE2kqFSNK1lynEuvwtYDpfhhcwGmjfTG2AG2T8S2xzVLqx/WHAfPKlCt0qJTG+sGV2M7OyGirRjrk8qIzMYgmRupWLScS/mDxyg8fAqPdx9C1iff49LL/0b4l++i7b9eNdq26OQ5FCSnICdxO86OfhWqsgr0PLbJfGxKnc3s+owb1pixxca4b9++WLhwIWbNmoUbN27g888/x5EjR7BhwwY0b97coYndy63GnQdKDOpp25iSva5ZWv2w5tBogNIKDdxcmI99CQXA5CEeuJKlRGm5BgHeAgR4C+DrpZuu5eLER4C3AC5O9o2n2ZIbyVi0nsuyK9chu3gNrf5Z96OVHqzfCedA8xe7aHU2N0afsaOocxygR48e+PLLLxEXF4f79+8jMTERnp6e9ZEbxCIe3F2tH6og4Zql1Q9rDqEA8Hbno6ySec0SCXmQuPHxTHsnLHrDV7/8+yXdT+eezzhj0Ru+6BPlXO+5kYxF87kUuDhD5FP354nvYnkGA63OZjbLXqvRMl5owOKYcc1DSbVaLaqrq5GamooePXpAq9WCx+Ph/Pnzdifg6cFHaZnxz5SObZ0Q3FyE9FvGc5AtQco1e+hEPuImtMTEMVJcvvbkbi+2/bBuzjxUyI0PalQvZwgEPFy5xfwKmbJaixW/Gd/J5uHGR/wID1zJUuD4BTlyHjO7ek4yN5KxaD2Xvn27w6NjKAqP6WxuAlcXgMeDuuKpsXs+H63/b5rFWKSPsTF8jhw17OYoLDbGlh5KSoqZ433hLRHg6k05CopVEAl5aNNCjJ5d3VCl0GL9HmaTywGyrl9a/bBDY53QOkiIrHsqFJVp4CTioWMbIcJaipD9UIVjF5hfIFNrYHJOsW42hQfyitVWzTkmmRvJWLScy7BP30bVvYfgOzvBM6ojgiYOh6qsAhn/WgQAcAtthdhDG/Bo+36UX89GdXEJnIOaIWjSSLiHt8H9ddsRHG/6zi8anM2OzstaNJT0eJnC+jzjlIsV6BPtht7RbpC4CQBokV+sRnJqGXYflaGwhPlVVtKuXxr9sFn3VWjuK0D3jmK4ufCg0QL5xRrsPlGFQ+cUYGFyh0NyI32cNJxL6bQxEPv7AFotqu4+xL1VW3DrqzWQ39cNAVTlPMaDjbvg0ysGzcYMhNDDDarScpRevIasL37Aw017zDbGJI+R5s+RNdAyFswU1hvj1EuVSL1k/5QqgLzrl0Y/7OVbKly+xXzSvS0UlmoYSYWehmRupI+ThnN5uE0/i+9XFxYj/c1PbYoNsO9sNkdD8BnTAKfQ5ODgaJR8uon5l/l/prDeL2W/Z8zB0ZQpKCjAd999h2PHjqGwsBB+fn4YOHAg3njjDSKy86aMtjFdwCONOdfsm7OeKPZS04p0ir2RQbiUXmpSsVfjOjUVj4ZYphy4yZti9NrF0BBXVFapGTlwSeRVkxuJvGr2oTUW6TpG8lyu2Ge4vrKsEFuWTUSFLA+dek5Ch96hKHyUhY2/bsa+w+cw4c1NEIkN5UCz//aT01ovnnYjA8D0cf56Jef17Cq9krN3jATZ9+UmlZw1nnF7UKsb1o9+1nvGJBV7tMYCyCk0ac2L5li0ltm5gytRVvwAQ+K+Qlj0SP36wNaR2L9+Hi4c/QX/GPw6o1i0HiNAVslpDQ1tBJZ1hSZJxR6tsQBy2kVa86I5Fq1llnPzNIQiZ7SPGmGwvn3kcAhETrh2ejvjWLQeI0BWyWkNDe12aNZ7xiQVe7TGIgmtedEMrWWmVikhEDmBxzO8VZjH50Mocoas8D6qyovg4u5jJsITaD1GS9ii5LQGWu6sYwrrPeO6FHvenmIITdzX3pBikYTWvGiG1jLzaR4KRWUp8nMMx1nzczKgqNTdrVZWzGyKGa3HaA5blZzW0OjcFOZYu3YtkQSYKvYaciyS0JoXzdBaZl37TgePx8e+xLdw59oxlBU/xJ1rx7Av8S3wBTornaqa2cMVaD1Gc9QoOTfuzrdKyWkNGo2W8UIDrDfGJBV7tMYiCa150QytZSZtG4Oh05dBqajA7p9exS8f98ee1f+HFqHdEdKxny4/Z3dGsWg9RlPYo+S0Bo1ay3ihAZvHjEldqSwoUqJ1sBtEQp7RTyxrFXu0xiIJrXnRDM1lFtp1GNp2GYzChzegVFTAOyAErh6+2LLsBfD5Qnj6tWIUh+ZjrI29Sk5r0NAy/sAQm3vGT190sBWSij1aY5GE1rxohvYy4/MF8G/RAdK2MXD18EWFLB/5ORmQtutmNM/YHLQfI0BGyWkNWq2W8UIDFhvjyMhIREVFGS2RkZHIyyNjWzp0Ih8ajRYTx0gN1tui2KM1FklozYtmGlKZaTUaHNv+GTRaNWIG/ZPxfrQfIyklpzU4asz4+++/R4cOHRAZGalf3nnnHbvzZV2hSVKxR2ssgJxCk9a8aI5Fa5kpFRXYsmwC2nYZBIlPCyjlZbhxfi/y7qejx4i3ERwa2+CPESCr5LQGRzb4MTExWL9+PdGYrM8zBsgq9miNRUqhSXNetMYC6CwzgUAE/6Bw3EjbgwpZPoRiFzQL7owxr61Gqw69rUsKdB4jQF7JyRR1A7PLU9EYk1Ts0RqLlEIToDcvWmMBdJaZQCjG0OnLiMQC6DxGgLySkynW3PQhk8kgkxnPd5ZIJCaFTVevXkVsbCxcXFwQFRWFt956C8HBwXblS0VjzMHBwUEaaxrjxMR1WL58udH6OXPmYO7cuQbrhgwZgvHjxyMoKAh5eXn46quvMGPGDOzatQtubm4258v5jDk4OBol/7ekhPG2i1/jW9Uzrk11dTViYmKwYsUK9OrVy+o8a6BCocnjARNGSzFmaBCaBzijpFSJIyfzsXrjHZMT1S1pL2mI9bRGMChAjP7dJega4Y5AfxFEIh4e5VcjJU2GXYeKoFAafx/WKASZPHFDLAQ+fd0X/t4CHDpTiY37jKcx/fxRACbNv2uwLtBfiBcGeSFEKoa3RACBACgoUeNCRhX2HJWhpMz0s422LG1F9FzOXmr6Q7NivpfJ9QqlFu98Z/wg1RXzvYjXMVN6ySljmqNda1e0D3FDYDMnPMpX1KnQTEhIwMW/jiLz9j08zCtAc39f7FzxpcnttVotdhw8jp3Jx3H3wSOIREJ0Cm2DmRNGofeEGQDI1Nea4yRRXwFdnSVRXsAT7ag9WNMzZtLomoPH4+kf3GwPVAxTvDHziYd18477Og/rKClC27ib9bA2pFiDnvXU+1yPnSnV+1zjxgagV7TEpM/VGsb2d4eHq/Xzvn09hfCSCHDmaiWKSlVQq4GWgSIMjHVHz66ueO/rXMjKrbsIQrL8AeD/2zv3oKiuOwB/yz4Al0V5REXwhZGM+Bas6EQ0mdiYYLSkU7Q2asaaGRsfaaumtjbGYSYZrRaTqtV0tE6iJlalKL5GIcbER9RERIVKggpVq44vxFWW1+72jx1IgHW5u9xlz8r5Zu4/93J/nnP48fM+zvnuxau1HD3X8OOjnryXUbNdv54UU+/6NRq1is7JyMggNMTIMz278bDx158b8Zf1n5KV8yVD+sYx67WfU1lVza7Pv+I3S1awoVs8w4YN82of1c5XT8ZLDbx1079v3z6SkpIIDw/n7t27rFixgvDwcAYPHtyiuD4vxqI6iNWM5U2fa7fOOsYkBbM95yGTXnTPzFVwsZKCi5VN9l+4XMXvpj7F6MQQsg8rl7io7dQFuFNu45sLNW6d4+12eeL6zc3NJeTeZQAmz1tCRaXzr1t/X3qVrJwvSRrUl5V/nFu/uCp1TDITf7uYxYsXs3///ibniZyvarqR3cFbsymys7NJT0/HYrEQGhrK0KFD2bhxIyEhypauPw6fm0NEdRCrGctbPleNBl5/xUTBxWpOX1D+6frmuF3mUBoag91LD7WdunVoAyBQ79GpXmmXJ65fpW/aTxcUAZAyaniDVa4mYzuShw6ktLSUvLy8JueJnK9qupHdwW6zK97cYd26dZw4cYKzZ89y5MgRMjIy6N5d2bJ1V/j8ylhUB3Fr+GFb6nP9aVIwUZE61mxrmWxFr4MgQwB6vYaYTnomp4QBcKZImTGsDm+M2eA4PUPj26MN0GCusHG6qIbdRy1UuvH37U+u35paRy4EBhqaHAsyOPbl5+cDvRsc84d8bW38zWfs82LcnId1QHx7dDqNIsmJqLGc0VKfa2SHAH42OoTsrx5xt9xGRHvPb3KeH2ZieuoPAvNb92pZteUORSXuXW2rPWalN2rJ+66G2/dtBBmgb6ye0UMC6d1Vx18/NVOl8OmFt3+XatIzxrEo4nTBdyQnDqrfb7fbybvwPQA3b96kcTEWPV99gb+JglwW45kzXa+NX7duXYsboNTD+rDW+Zt9f4jljDqf68dZtzzyuU5NMXG7zMrBr12/DFLCNwUVXL9VQ6BBQ89oAwl922Eyul/c1R6z5Vsazgo59Z8art+2Mn5kMKOHBHLgpLL/LLz9u1ST4YP70TMmiswDh4kMa8/oYUOorKrmsz05XL5yHQCLpekdi+j56gueqCvj/Px8oqKiSElJYeDAgV55O1lZZSUsqOktGXjmIBYxVmNa6nNN6h9IfC8Dyzbe92hmQWPulVu5V+74I/220MLJ8xW8/1YUgQYNOw8pvwry5pjVkfNNFS8ND6JfrF5xMW6NdqmFTqtl5Z/mkr5mI2u2/Js1WxzfwXu6ewxv/upV/vbJdqcvikTOV1/hb0soXBbjY8eOcezYMfbu3cuePXsYNWoU48aNo3fv3q5OcwtRHcTe8sO21Oeq08KkF02cL66m/KGNjmGOqUIdQh1/cMGBAXQM02KusGGp8iwZr9yoofR/1YwZYXKrGLeGU9dmg/JHNozByqfy+Yvrt47OkRH8/d353Lxzlxu37tLeFEJs1y7sOHAYgNjYWDje8BxR89WXWB9zpyAqLu9FtVotycnJLFu2jG3bttG9e3emTJnC5s2bVWuAqA5ib/hh1fC56nUaQo0BDIwLZOnciPpt4euOl24jBgaxdG4EyUOCPIpfh0GvIaSde48qWsOpq9NCWEgA5grlhcUfXL/O6BwZweD4OGK7Op4jf33mPAEBAU5XeYmar77kifIZA1RXV3Pw4EHmz5/Pli1bmDJlCmPGjFGtAaI6iNX2w6rlc62usbNmW3mT7ZO9ZgDOF1exZls5+d81/1yvvcn5r79vr0C6dtZT/F/3XuCpOWbGIOdXvq88G4RWq+H8JeVzj0V3/Srhq2/zOZZ3ngkTJhAdHd3kuKj56kvsNpviTQRcPqZ4++23KS4uJjk5mdmzZxMXF6d6A0R1EKsZS02fq9WG0znFjtkUJm6VWRXPOZ7xagRhoVoKLlZyp6wWvU5DbIyBEYOMWKrsbNrt3kIUNcdsbFIgPbroKL5Syz2zjUC9hr6xOp7ppqfkei1fnlH+H4UIrt+dO3dyKc/xbKHsgZnaWiv/zNwLQNRT4byUPLz+Z99b+zF2u53ePboSaDBwrugiB46eJL5XDxYtWuT1PqrtH1bTjewOonxoVCkui3F2djbBwcGUlpY2ECnb7XY0Go3TyeeeIKqDWK1YvvK5Nsex/EckJxgZmWAk1KgF7Nwus5J7wkz24Qfcve/+m3e1xqz4ai2dI7QM62vAGKzBZofbZTayj1j4/Nsq3J0U4GvXb2ZmJqdOnWqw7x//2gXA4Pi4BsU4/uke7Mw9whcn86ittRLT+SneSBvPpHEvYDI9fr6wqPmqtptaKaI8flCKy2JcVFTUKo0Q1UGsVqzW8LneLbcpkgr9mBNnKzhxtuVT436MWmN27lIt5y6pt7jA167fTZs2UXbWucSoMaljRpE6ZpTb/4ao+aq2m1opNj97gefzRR8SiUTiDWx2/yrG0mcskbQCH330EYWFhRQWFnLt2jWio6M5dOiQ059duHAhWVlZTo99+OGHjB071ptNfWJInV2s+GezVqs3XddThPAZv/XGD+q/E6fvOdR/47pwtrDcqfrPlYPY01jOPKy5nyXWq/9692xHhcXarLdW7Vhq9LGun2q0q+4cNWOpmRei5lhGRgYhplB69oqj7H451TVWzhU7f6x074Fjhd2ceX9ucmzAgAGqtauubaLGailP1Aq81kBU7SWoq/5TK1Zb6COImxeexsrNzaWs0jGj4PdvTqWysnkJU/JzLzbZ16WLc/uaCH30dix38bebfqnQdIGa6j+1YrWFPoK4eeFpLE8+Vmm326moeIRNwTxYEfro7VjuYrPZFG8i4PMrY3/TXvqattBHEDcvWnP8p6WNxWKpQKfT06ffQH45ZQYDejsvXqL20Zf5arP6XvzkDj6/Mm5O/RfW3oBOp8xDoGYsUWkLfQRx86I1xr9DWAQpE9J44815LFj0Hq+mTeFycRHv/GE2x48fd3qOqH30Zb56Sy7vLXx+ZexP2ksRaAt9BHHzojXG/7XXG6prfzI8mWdHvcCCudNZsmQJBw8e9Gq7RI3lLqIUWaW4LMarV69+7DGNRsOsWbNa3AB/0V6KQlvoI4ibF74a/6jorgwf+RyHc/dTUlLi1XaJGstd/G2escvHFO3atWuyaTQaMjMzWb9+vSoNuHOvmvahevROblU80V6qFUtU2kIfQdy88OX4d+zo8DmUlTV1hojaR1+Ol789pnBZjKdPn16/paWlUVlZSWZmJi+//DK5ubmqNEB07aVotIU+grh54cvxv3H9KgCRkZFebZeosdzF36xtzb7Au3//PitXrmT8+PFYrVaysrJYsGABERERqjRAZO2liLSFPoK4eeHt8a+stFBd3dRIV3Lpe74+ephevXrRrVs3r7ZL1FjuYrNaFW8i4PKZ8bJly8jJySEtLY3du3djNBpVb4Co2ktQV/2nVqy20EcQNy88jbVz507OnHcsz33w4D61NTVkbv0YgMiOnRj1vGOJ843r13j/3fkMTRpJVJcYgoKCKS25yBc5+wjQBpCeni5sH70dy12eKIXmxo0bMRgMrF27tsHHR9uKQlNN9Z+asdpCH0HcvPAkljOF5tbNjvcu8f0G1RfjDmHh9B+USOH5Mxw9nEN1dRUdwiMYMfJ5Un/xGomJicL2sTViuYM3Hz/YbDY++OADduzYgcViYciQIaSnpzsV/ytFKjRdoKb6T81YbaGPIG5eeBJr06ZNj3VR/JiwsAjmznun1drlb7HcwZsv5tavX8+ePXvYvHkznTp1YunSpcycOZNdu3YREODZ8g2fL/qQSCQSb2C32xRv7rJ161ZmzJhBbGwsRqORBQsWUFJSwunTpz1ur1RoSiSSJ5LHGfycsW/LYB48aPol9NDQUEJDG84EMZvNJCYmsn379nqLHkBKSgoTJ05k6tSpHrXX5yvwJBKJxBu4o+FctWqV00Vus2fPZs6cOQ32PXzomI7XuEibTKb6Y54gi7FEImnzTJs2jdTU1Cb7GxdcgJCQEMBxhfxjzGZz/TFPkMVYIpG0eZw9jngcJpOJ6OhoCgoK6N+/P+AoxFeuXKFPnz4et0G+wJNIJBI3mTRpEhs2bKCkpISKigqWL19Ojx49SEhI8DimvDKWSCQSN5kxYwZms5nJkydjsVhISEhg7dq1Hk9rAzmbQiKRSIRAPqaQSCQSAZDFWCKRSARAFmOJRCIRAFmMJRKJRABkMZZIJBIBkMVYIpFIBEAWY4lEIhEAWYwlEolEAGQxlkgkEgH4PyOTD5a1MK7qAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g3LhE1WNLnsh"
      },
      "source": [
        "## **Model 5** : LSTM (variable-length inputs + word2vec word-embeddings)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DYTdRjFrLzjY"
      },
      "source": [
        "class LSTM_w2vec2_variable_size(torch.nn.Module) :\n",
        "    def __init__(self, vocab_size, embedding_dim, hidden_dim, vec_weights, freeze_emd=True) :\n",
        "        super().__init__()\n",
        "        self.embeddings = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)\n",
        "        self.embeddings.weight.data.copy_(torch.from_numpy(vec_weights))\n",
        "        if freeze_emd:\n",
        "          self.embeddings.weight.requires_grad = False ## freeze embeddings\n",
        "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first=True)\n",
        "        self.linear = nn.Linear(hidden_dim, len(tag_to_ix))\n",
        "        self.dropout = nn.Dropout(0.2)\n",
        "        \n",
        "    def forward(self, x, s):\n",
        "        x = self.embeddings(x)\n",
        "        x = self.dropout(x)\n",
        "        x_pack = pack_padded_sequence(x, s, batch_first=True, enforce_sorted=False)\n",
        "        out_pack, (ht, ct) = self.lstm(x_pack)\n",
        "        return self.linear(ht[-1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KxYgn7HtLnsj",
        "outputId": "a5fa4042-1706-4751-c40f-035184305cc1"
      },
      "source": [
        "w2vec_variables_notfreeze = LSTM_w2vec2_variable_size(vocab_size,\n",
        "                         embedding_dim = w2vecembedding.vector_size,\n",
        "                         hidden_dim = 50,\n",
        "                         vec_weights = pretrained_weights,\n",
        "                         freeze_emd = False)\n",
        "train_model(w2vec_variables_notfreeze, epochs=100, lr=0.01)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train loss 2.564, val loss 2.517, val accuracy 0.179\n",
            "train loss 0.385, val loss 2.252, val accuracy 0.384\n",
            "train loss 0.006, val loss 3.000, val accuracy 0.397\n",
            "train loss 0.002, val loss 3.227, val accuracy 0.387\n",
            "train loss 0.001, val loss 3.249, val accuracy 0.403\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 664
        },
        "id": "EpY7cb3AOfsI",
        "outputId": "33dd9b47-2f00-4b31-f5b8-1ea24f56eef3"
      },
      "source": [
        "print(\"***Model 5 : LSTM (variable-length inputs + word2vec word-embeddings)***\")\n",
        "pred_vs_true_m5 = matrix_pred(w2vec_variables_notfreeze, val_dl)\n",
        "diagnostics(w2vec_variables_notfreeze, val_dl)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "***Model 5 : LSTM (variable-length inputs + word2vec word-embeddings)***\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           A       0.50      0.27      0.35        22\n",
            "           B       0.38      0.42      0.40        12\n",
            "           C       0.27      0.23      0.25        13\n",
            "           D       0.45      0.68      0.55        22\n",
            "           E       0.19      0.15      0.17        20\n",
            "           F       0.32      0.34      0.33        29\n",
            "           G       0.31      0.21      0.25        19\n",
            "           H       0.57      0.55      0.56        65\n",
            "           I       0.30      0.17      0.21        18\n",
            "           J       0.47      0.38      0.42        21\n",
            "           K       0.42      0.44      0.43        61\n",
            "           L       0.30      0.18      0.22        17\n",
            "           M       0.28      0.51      0.37        37\n",
            "           N       0.60      0.47      0.53        19\n",
            "\n",
            "    accuracy                           0.40       375\n",
            "   macro avg       0.38      0.36      0.36       375\n",
            "weighted avg       0.41      0.40      0.40       375\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEDCAYAAADgNmW7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd3xUVd7/3zOTmbTJpEACJLSAGLoEooAoJSJFmqAoRIoNfR4BVwR/z7q666prWXEtKxYWXOmiEQUEUUCKgIQmNSQQBJGeBJIMKTOTKb8/xgTCTCZTzmRukvt+ve4Lcss3n3PmzDfnnnvO5ypsNpsNGRkZGZmAogy0ABkZGRkZORnLyMjISAI5GcvIyMhIADkZy8jIyEgAORnLyMjISAA5GcvIyMhIAEkl440bN5KUlMSvv/7qU5wOHTowatQoRo4cyejRo/nll198ipeXl8eMGTMYOHAgY8aMYcqUKZw6dcprXcOGDWPkyJH897//xWq1eq2rIl7F9p///EdYrLNnz3odKz8/n5kzZ3LXXXcxZswYHnzwQTZs2OBxnOTk5Co/f/3117zyyite66oubqDjXX/91q1bGTx4MOfOnQuYngqSkpKYNWtW5c9ms5levXrx5JNPehXrzTffrPz5008/5YMPPhCis74QFGgB17NmzRp69OjB2rVrefrpp72OExISwqpVqwDYtm0b77zzDkuWLPEqls1mY9q0adx77728++67AGRnZ3P58mUSExO91nX58mVmzpxJcXGx12W9Pp6viIpls9mYOnUq9957L//6178AOHfuHJs2bfI5dn1n586d/OMf/+DTTz8lISEh0HIICwsjJycHg8FASEgIO3bsoEmTJl7F0mg0rF+/nieeeIKYmBjBSusHkukZl5SUsG/fPl577TXWrl0rLG5xcTE6nc7r6zMyMggKCmL8+PGV+9q3b09KSopPuho1asSrr77K0qVLqU/rbjIyMlCr1VXqKyEhgYkTJwZQlfTZs2cPL774Ip988gktW7YMtJxK+vXrx5YtWwBYu3Ytw4YN8ypOUFAQDz74IAsXLhSorn4hmZ7xjz/+yJ133kliYiLR0dEcOXKEzp07exXLYDAwatQojEYjeXl5PjWAnJwcOnXq5PX1rmjRogUWi4XLly/TuHFjj6+vKGcFTz75JPfcc49XWq6P1bx5cz788EOv4uTk5NCxY0evrnWlCaCoqIjU1FQhsaWEyWRi6tSpLFq0iLZt2wZaThXuuecePvroIwYMGMCxY8e477772Ldvn1exHnroIUaOHMnjjz8uWGX9QDLJeO3atUyaNAmwN4C1a9d6nYyvv+Xev38///d//8eaNWtQKBTC9EoBKQ5T3MjLL7/Mvn37UKvVrFixwidNX3/9NUeOHBEtMeAEBQWRnJzMV199xYsvvhhoOVVo3749Z8+eZc2aNfTr18+nWFqtllGjRrFo0SJCQkIEKaw/SGKYorCwkIyMDF588UVSU1P59NNPWbdunZDb9+TkZAoKCrhy5YpX17dr147MzEyfdTjjzJkzqFQqGjVq5Jf4gaBdu3YcPXq08ueXXnqJBQsWUFBQEEBV0kapVPLee+9x+PBhPvnkk0DLcSA1NZW33nrL6yGK65k8eTIrVqygrKxMgLL6hSSS8Q8//MCoUaPYvHkzmzZtYuvWrTRv3py9e/f6HPvXX3/FYrEQFRXl1fW9evXCZDLxxRdfVO7Lzs72WduVK1d46aWXeOihh+pVj71Xr14YjUaWLVtWuc9gMARQUd0gNDSUuXPn8u2335Kenh5oOVW4//77mTp1KklJST7HioqKYsiQIXz11VcClNUvJDFMsWbNGqZMmVJl36BBg1izZg233nqrx/GuH2u02Wz885//RKVSeaVNoVAwZ84cXn/9debNm0dwcDAJCQn85S9/8VqX2WxGpVIxatQoHnnkEa90XR+vgjvvvLPKVKRAoFAo+PDDD3njjTeYP38+MTExhIaGBlyXvzCbzWg0GiGxoqKimD9/Pg899BAxMTHcddddXsUpKyujb9++lT8/8sgjPrWzpk2bVg4hiuDRRx9l6dKlwuLVFxSyhaaMjPdkZ2fz4osvyj09GZ+RRM9YRqYu8vnnn7N48WKv7pJkZG5E7hnLyMjISABJPMCTkZGRaejIyVhGRkZGAsjJWEZGRkYCyMlYRkZGRgLIyVhGRkZGAsjJWEZGRkYCyMlYRkZGRgLIyVhGRkZGAsgr8GRkZOola9XuGxsNKz/mRyXuUavJ+JkPip3uj45QMG10KMEaBRlHy8krsBISrCC+kZLs3y3szzFXOf+96VoAhk85WmX/5DFxDBsQze6DxWSfLMVigS5JYfS9NZJTZwzMfOMUpvKqCw7XzOsoPNakv16osS7GDY5gwK1hhAYr2ZBRwuK1+irHF73aDIA7Rmx1uPZPU9oydmRztv6cR8a+K7RqEc79w+M5mFnEM389hLM1ldu/7edQRl/K6UyXN9q2f9tPsrFATP1XxBo43tHpb+PnKeivmsn5rZR2iWGUllmY8PRhp2WoOF+UrgptT77p3F527p9jyDlTzrYDxir7LRbYm21yer5IXQ0NSfSMJwwKQamEt5aVoi/1fnX2jn160tflU1p27SWf67YWcD7XxLhhsQy6I4o1m93z1RUZ60ZaNQticO9wvlh/lbShnr0SKrFlGPcNT2DLz3m8+Ma15HrhUhkznmzHwL5xbNia63Y8keUUqa0hxAKY+KdDXMi1J7Z5b3UiNMS7kUPRuirIK7SyK9Mx8QZalzso1HXLmjbgY8Zt4pW0jVex6Zdy9KU2lEpQe/kn4sRpQ5WkUsG2PfZeZ6uE4IDEuh6FAh4dFcXhE0b2HvXc53dg3ziUSgVfrqr69uBvf7hAmcHCoP5xHsUTWU6R2hpCLKAyEfuKaF3Xo1JCsFp6umpCFapye/OEjz76iIEDB9KjRw969uzJY489RlZWVuXxpKQkunbtSnJycuV27FjNwyAB7xl3bG2XUHDVyuPDQ+jQSoVKqSC3wMoPe0zsO2auIULNNI62t6QCvSXgsYbcHk58rIoPlnvXq27fLgKLxUbW8arDGqZyGzkni2nfLsKruDfiTTlFamsIsUTiL13dkzT07KRBpVSgL7GyL9vEyp/KMBjdu4MNZH0pg/zTMx46dCgPPfQQkZGRmEwmlixZwpQpU/jpp59QKu3923nz5tGzZ0+P4nrcM967dy8vv/yyp5dVS1yUXcKDqcGEBStYtsHIso0GLFYbEweFcFsH3/5eKBUwblhjzGYbW3cVBTRW4ygVY1K1rNxSTH6hd8m8cYyGIn055WbHL0P+FSPRkRqCfGyE3pZTpLaGEEsk/tB16ryZNdvLmPtNMZ+tKebY6XIG9AjhuYci3O4pB7K+FGqF25ter+fs2bMOm16vd4ibmJhIZGRk5c9KpZK8vDyuXr3qk163Mt3Ro0f59ttv+eGHH0hISGDQoEE+/dLrCf7jJQnGcvjwmzIsf9wxHz5p5q+TwhnWW8OeLDPejiRPGdeUDjeFsfDrS5y75Nstoa+xHhkZSW6Bhe93lHitISRYRbnZcVgBwGSy/nGOkmKz93cB3pZTpLaGEEsk/tD15qKqiSjjiIlzeRbu7RdGakoI63bWPMwWyPrypGe8cOFC5syZ47B/2rRpTJ8+3WH/li1bmDVrFlevXkWhUPDII49USdAzZ86kvLyc+Ph4xo8fzwMPPFCjhmqT8alTp1i7di1r1qwhOjqae+65B5vNxuLFi90tn1uU/zEK8ctxc2UiBigzwpFTZm7roCYuWsGlAs/T8YRRsYxIjWHd1gLS1132SaevsW6/JZRObTW8/unlKuX0FIPRQnSI89f8aDTKP87x/hf4Uk6R2hpCLJHUlq4fdhkY1ieULm3VbiXjQNaXJw/wJk+ezOjRox3263TOH7D379+fvXv3UlhYyMqVK2nWrFnlsQULFpCcnIxSqSQjI4NZs2ZhNptJS0tzqaHaZDx06FBSUlKYO3curVq1qvwloikqtidZZ7MoKvaFBivAw75x2ohYxg2PZcP2Aj5cUvNUM3/GClJB2pAIDuUYKSy2Ehdjf2AQrbP/GxaiIC5GRXGplVKD63LmXzHRukU46iCFw61f45hgCopMmJ3cErqDr+UUqa0hxBJJbemyWqGo2Io2zL0RzkDWlyc9Y51OV23idUVUVBSTJk3i1ltvpU2bNrRr147evXtXHu/bty8PP/wwq1evrjEZV1ujc+bMITY2lkmTJvHiiy+yc+dO/PFSkNOX7LcnUVrHiosKt+8rLvM8EaeNjGXjz4X8e5HvidjXWBq1Ap1WRbekEN6eEVe5vfBYIwD6dAvj7Rlx9OsRVmOs7JyrqFQKOtxcteFo1AratdFy7ITzudw1IaKcIrU1hFgiqS1dQSqIjlCiL3GvNxvI+lJplG5vvmC1WjGbzZw+fdrpcaVS6VburFbFwIEDeffdd1m3bh09e/Zk4cKFla+X3759u/fKb+DwSTMGk42UpCA01z0U0IUp6NImiNwCK/lF7ifjccMbkzYylk07C3l/wXmnE8prO5bRZOOD5QUO24LV9odjB48b+GB5Afuza77t+3FbHlarjQdGJVTZP2JwM0JDVKzfcsljfaLKKVJbQ4glEtG6wkOc9ypH9Q1FpVJw6ER5QHR5gkKpcHvzhEWLFpGXlwfAlStXePnll9FoNHTr1o3MzEwOHz6MyWTCbDazY8cOPvvsM4YNG1Zj3Bof4IWFhTFixAhGjBhBUVER33//PfPmzeOOO+7wqADVUWaEVduNPJgawoyxoew6akalgj6d1ahUsGKrseYgfzCsfzQTRsWRe9nEgawS+t0WWeV4od7MgSz3Hp6JjGWxwp5Mx0TbOMo+TJF7xeL0uDNOni7h67XnuX9EAq8935Gd+67QunkY949IYP/hQo8n0Issp0htDSEWwMA7YmgSa5/LHaULIkil4KHR9vHHS3lGNm53vjrO37ru6RNKm/ggjp0u54reSrBGQee2atq3UnPynJnN+wLTXj1BofLPMoqMjAzmzp1LSUkJWq2WLl26sGDBAho3bsyhQ4eYPXs2Fy9eRKVSER8fzzPPPMP48eNrjOvRvLHIyEgefPBBHnzwQa8L4oydmWZKDGWkdtcwtJcGmw1+u2hh8XoTpy64P7jfLjEUgLhGGp59NMHh+OFjJW4nFpGxRPPv+Se4mGtg5OBm9L61EUX6cr5ac55Pl57yuFcrupwitTWEWEMHxHJLx6pzbR95wP45HDx61e1kLFrX8d/LadZIRa8uwWhDFVitkFtgYeXWUjbsNuDJ5AeRujxBqfLPlLmPPvqo2mOpqamkpqZ6FTfgiz4qOPSrhUO/lvkU473PzvPeZ+eF6BEZqzryCy1u+VjciNUKy1eeZfnKsz5rEF1OkdoaQqyZr4ozqBGp62BOOQdz3BuKqAmRujzB0+GHQCOZZCwjIyMjEl8fzNU2Cps/pkjIyMjIBJh9A/q4fW6PzTv8qMQ95J6xjIxMvUQepnCBMz9X8N7T1ZkPqzcerCDWt1ZqHrgV14j0mhWpS6o+yyDWg1iqfsZSjeUr/nqA5y8k0TMW5elaga8erCB74AZSF0jXZxnEllOKn6VUY3lKXesZS2KEW5Sn6/X44sEKsgduIHWBdH2WQWw5pfhZSjWWpyiUSrc3KSCJnrFofPVgBdkDN5C6XBFon2WpItX2Gsi6V6mlkWTdpW6pdQMRHqwge+AGUld1SMFnWapItb0G1M/YT8uh/YXHPeMrV64QHR2NQiGNAtyICA9WkD1wA6mrOqTgsyxVpNpeA1n3Uhl+cBeXag8cOMDEiROZNm0aR48eZfjw4YwYMYLbb7+dn376qbY0+swPuwyUm210aet+19hgtKAOcl493njgioolEqnqcoavPst1pZzeItX2Gsi6r2s9Y5fJ+JVXXuHJJ59k2LBhTJ48mX/84x/s2LGDJUuW8M4779SWRp/x1IMV7D6skTo1aie3UN544IqKJRKp6roRET7LdaGcviDV9hrIuq9XydhisXDHHXcwdOhQGjduTLdu3QBo27ZtrYgThacerCB74AZS1/VIzWdZqki1vQay7pVBKrc3KeAyGSuvG3MJCQmpckyKY8aiPFhB9sANpK4KpOizLFWk2l4DWfdKlcLtTQq4fICXnZ1N9+7dsdlsGI1GunfvDoDNZsNkEjfXUpSnqygPVpA9cAOpC6TrswxiyynFz1KqsTxFKsMP7uIyGWdlZdWKCFGeriI9WEH2wA2kLin7LIssp1Q/S6nG8oS6NptCEos+RHm6ivRgBdkD11NE6pKyz7LIckr1s5RqLE+oVz1jGRkZmbpKXUvGsp+xjIxMveTMU/e5fW6Lj1b4UYl71GrPuDp7Q4UCxo5MYNSQeJrGhVBYZGLz9jzmL/3N6YRwV/aG40c15abWYdycGE6zJsFczDO6ZUk4dsapKvvjY9XcPziKxOYaYnRBqFSQX2Bmf1YZqzYXUejEGyH93UQAB0vI+CYaBvSMJLlTOM1iNajVCi7mlbN9r55VGy9jNDn+PVwzryPgvM48rS+o3t5QZCxv4kk5Foip/4pY1Vm+OsNgsvGndxxd6VxZvnr7WYpor1C9ham3unxFHjP2gqcfv+Z3uvybM3a/0xEJtGujrdbvtDoeG9e80h82PNz7+YMxUSqidSp2HyrlcpEZqxVaNlMzsHcEtyeH89zb59AXuzdv+e4+UZXevFt2FVV6804aHcedKTqn3ryuEFlfImNJWZtUYwHV+m97iihdUm6vHuGn6bcfffQRX3/9NQUFBQQFBdG5c2dmzZpFhw4dADh69CivvPIKWVlZREdH8+ijjzJp0qQa4wY8GUvVa/ZIjoEjORcd9h/91cDMh5vQ/7YIVm9yz6xGqt68outeqtqkGqsCqflvS7W9eoq/xoyHDh3KQw89RGRkJCaTiSVLljBlyhR++uknSktLefzxx0lLS2PhwoVkZWXxxBNPEBcXx5AhQ1zGDXg/Xspes87ILzADoA11v+qk6s0ruu6lqk2qsa5HSv7bUm2vnuIvP+PExEQiI6/Ne1cqleTl5XH16lXWr1+PUqnkqaeeIjg4mG7dujF27FiWLVtWY1yXPePTp0+Tn59Pjx49quzft28fsbGxtGzZ0qNCOEPqXrPqIAUhwQrUQQqaN9UwYXg0AL9klfocO9DevKLrXqrapBqrAqn5b1dHoNurp3jSM9br9ej1eof9Op0OnU7nsH/Lli3MmjWLq1evolAoeOSRR4iMjCQ7O5uOHTtWWb3cuXNn0tPTa9TgMhm//vrrPPvssw77tVotr7/+Op988kmNv6AmavI77doxkqAgRcCMXO7qpeWx+xpX/px7uZz3F+eSfdLo4qqa8Zc3ryf1JbrupapNqrHA7r+9L9tEboGF0GAFnduoGdAjhHYtgnhrsR6jm9Pm/f09kkJ79RRPPCcWLlzInDlzHPZPmzaN6dOnO+zv378/e/fupbCwkJUrV9KsmX3VZHFxMRERVf/A6HQ6iotr9uBwmYzz8/NJSkpy2J+UlMS5c+ecXOE5Uvea3X24lHO5FwjRKElsriGlUxg6re/GIlLw5hVd91LVJtVYIE3/bWdIob16iic948mTJzN69GiH/c56xdcTFRXFpEmTuPXWW2nTpg1arZbLl6tavOr1erRabY0aXCbjq1evVnvMYHDf58EVBqOF6BCN02NS8Jq9UmThSpG9oew5UkrGwRLefDYBjVrByh/d7yFcj6/evKLqS3TdS1WbVGNVxw+7DAzrE0qXtmq3k7E/dUmlvXqMB2PB1Q1HuIPVasVsNnP69Gnat2/PunXrsFqtlUMVmZmZtG/fvma5rg527tyZL7/80mF/eno6nTp18kr4jdQ1r9nfL5Rz6qyRwX28++Ck5M0ruu6lqk2qsaoj0P7b1yOl9uopCoXC7c0TFi1aRF5eHmB/89HLL7+MRqOhW7duDBo0CIvFwscff4zJZOLQoUOkp6czfvz4GuO6/LT/8pe/8PXXXzNx4kTefPNN3nzzTSZMmMBXX33FCy+84FEBqqMues1q1AqPvigVSM2bV3TdS1WbVGNVR6D9tyuQWnv1FH/NpsjIyODee++lW7dujBw5kvz8fBYsWEDjxo3RarXMnz+fn376iZSUFKZPn87UqVMZOnRojXFdqmjcuDHLly9n6tSpJCQkkJCQwNSpU/niiy+IjY31qADVIVWv2agI5+PCnW4KoWUzDTmnPXuAJ0VvXtF1L1VtUo0lVf9tkGZ79RRFkMrtzRM++ugjduzYwYEDB9i+fTsff/xxlZGCjh078sUXX3Do0CG2bt3q1oIPcHPRR69evejVq5dHgt1Fql6zU+5vRJROxZEcA3kFZjRqBW2aa+iTrKXMaGXRqrrvzSu67qWqTaqxpOq/LdX26il1zSgo4CvwQJpes9v3l9AvRUvfFC06rRJskFdgZsNOPas3FZFf6P7TXyl784r2mpWqNinGkqr/tpTbqycoFAFf0+YRkkjGUvSa3XmghJ0H3G9wrpCyN69or1mpapNiLKn6b0u5vXpEHesZyxaaMjIy9ZLCf05z+9yo/3Nc8FHbSKJnXJ+YO3cumZmZZGZmcvbsWRISEti0aZPTc//85z/zzTffOD32/vvv12gsIiMjUz3ymLELnPkPg91TuML2sl1iGKVlFrc8iJ3F8zaWMx/WP025Zv2Xse+K3fpveDwHM4ucWv9t/7Yf77zzDuERkbRqk0R4oR6DycrPWc4Xz+QW2m9RpzzzisOxrl27CitjxTVSjVWdB7E39S8yFohrFyC99lqhTWQske3CVxQq31fK1iaS6BmLsr0UGctb67+NGzdypjgKgBeffgBDWVmNv+v2/vc47IuPr95ARYr1JTqWVG0vpWr52hDK6DF1zFxeEmpF2l6KiuWt9V+LFi08/l02m42y0mKsVvcm+UuxvkTHkqrtpVQtXxtCGT3FXyvw/IUkesZSpDat/55K64+hrISgIDU3d0pmzENPcXuH3sLi10WkanspVcvXhlBGj6ljPWM5GVdDbVj/RUY3YtDINFq37UBwcCi//3acDd9+zht/eZy2sfO4/fbbfSlCnUaqtpdStXxtCGX0lHr7AO/KFftiiZgY5y9QrG/UhvXf2ElVfVK79+pPr75DeGlGGn//+99Zv36917HrOlK1vZSq5WtDKKPH1KdFHzabjTlz5rBkyRJsNhs2mw2VSsWECROYNs39OXx1kUBZ/zWNb8ltfe5m+6ZvOXXqVM0X1FOkanspVcvXhlBGT6lrsylc/ulYsGABv/zyC1999RW7d+9mz549pKens3//fhYsWFBLEgNDIK3/GsXZvTQKCtx76WN9RKq2l1K1fG0IZfQYpcL9TQK4TMarVq3iX//6V5UZAi1atGD27NmsXLnS7+ICSSCt/y5dOAPYXfMaKlK1vZSq5WtDKKOn+MtC01+4VGE2m52OEcfExGA2m/0mSgr42/rPaCij3ORow3n6ZDZ7d2ykbdu2Ql74WleRqu2lVC1fG0IZPUahcH+TAC7HjNXq6t8d7uqYp4iyvRQZy1vrv5UrV5Jx0D7We7WoELO5nNVfzgegcWwzbh8wzK7l/O+88+rTdO/ZnybNWqAJCeXMqRy2/bgapVLJK684rsoTXUYpx5Kq7aVULV8bQhk9RiI9XndxmYyzs7Pp3r27w36bzYbJJG4ityjbS9GxvLH+W7FiBbt3766y75tl9rdoJ3XqXpmMI6Mb0bHrbWQd3svOrd9TbjIQGd2Y2+64m+H3PUJKSpdaKaNUY4E0bS9Fxwp0e62NWKLbhbvUtQd4LpNxVlZWrYgQZXspOpY31n+LFy+u1ovieiKjG/PEjFe90iXV+hIZC6Rpeyk6VqDba23EEt0u3KY+TW2TkZGRqbNIZJaEu8h+xjIyMvUSw+f/dPvckPH/50cl7lGrPePq7A0VChg7MoFRQ+JpGhdCYZGJzdvzmL/0N6eTy11ZEo4f1ZSbWodxc2I4zZoEczHP6LUlobe6hk85WmV/fBMNA3pGktwpnGaxGtRqBRfzytm+V8+qjZcxmhz/Hq6Z1xGAsTOqLvyIj1Vz/+AoEptriNEFoVJBfoGZ/VllrNpcRKHe+aqo9HcThZSxopw3ltGXcoqMVZ1Nq7ftQqptTISuCm3PfOB8qlpYMAxM0dClTRBRWgXGcrhw2cK6XSZOnnfU9t50rVBdPuOnnvHs2bPZsmULFy5cICwsjL59+/Lcc88RHR1deU5SUhLBwcGorhu3Xr58OUlJSdXGlcQwxdOPX/NOXf7NGbt36ogE2rXRVuudWh2PjWte6Z0aHu7bAL4oXXf3iWLYgGh2Hyxmy64iLBbokhTGpNFx3JmiY+YbpzCVuxcsJkpFtE7F7kOlXC4yY7VCy2ZqBvaO4PbkcJ57+xz6YvdXR4mse5HlFBkLxLYLKbYx0bqiIxRMGx1KsEZBxtFy8gqshAQriG+kJDJcCbjfxkTq8gg/jRmrVCpmz55Nu3bt0Ov1PPfcczz//PN88sknVc6bN28ePXv2dDtuwJOxVL1TRerasU9P+rp8SsuuNeB1Wws4n2ti3LBYBt0RxZrN7q22O5Jj4EjORYf9R381MPPhJvS/LYLVm4rciiW67kWWU2QskKZvs1TbPsCEQSEolfDWslL0pb6NZAbMz9hPsymeffbZyv83atSIiRMnMnPmTJ/jBvxxo1S9U0XqOnHaUCWpVLBtj92isFVCsG9isQ9VAGhD3f9IRde9yHKKrjMp+jZLte23iVfSNl7Fpl/K0ZfaUCpB7UO3LVB+xiiUbm96vZ6zZ886bHq9vsZfs3PnTtq3b++wf+bMmfTs2ZPRo0fz5Zdf1hgn4D1jqXqn1oauxtH2hTMF1YzzukIdpCAkWIE6SEHzphomDLePV/2SVep2jNqqe1/K6c9YgUaqbb9ja3taKLhq5fHhIXRopUKlVJBbYOWHPSb2Hasjq289GDNeuHAhc+Y4vpR02rRpTJ8+3ckVdr777jvS09NZsmRJlf0LFiwgOTkZpVJJRkYGs2bNwmw2k5aWVm2sgCdjqXqn+luXUgHjhjXGbLaxdZd7wwrXc1cvLY/dd827IvHknn8AACAASURBVPdyOe8vziX7pOMS6+qojbr3tZz+iiUFpNr246Lsd1cPpgaTV2hj2QYjKhUMSFYzcVAIKqWB3Vl1ICF7MGY8efIkRo8e7bBfp9M5OdvO2rVr+fvf/87HH39Mp06dqhzr3fvayyH69u3Lww8/zOrVq10mY5dq582bV/n/devWVTn2zjvvuLrUbdz1Tq1t/K1ryrimdLgpjKWrczl3yfPbuN2HS3nl4wu89ekl0n8ooKTMik7r2RhZbdS9r+X0VywpINW2H/yHe6axHD78pox9x83szjLz7xVllBpsDOutoU7M4PXAm0Kn09G8eXOHrbpknJ6ezssvv8wnn3xCr169apSiVCqpaRaxy0/6u+++q/z/f/7znyrHtm3bVqMAdzAYLaiDnMsItD+sv3RNGBXLiNQY1m0tIH3dZa9iXCmycPi4gT1HSvny+0LmLMtjwogY7r0r0u0Y/q57EeX0RyypINW2X/5Hp/eX42Ys1/36MiMcOWUmMlxJXHQdSMdKpfubByxatIi3336bTz/9lB49ejgcz8zM5PDhw5hMJsxmMzt27OCzzz5j2LBhLuPWaC7v7P/OfvaW/CsmWrcIRx2kcLhdC7Q/rD90pY2IZdzwWDZsL+DDJRdEyeX3C+WcOmtkcB8dK3907xben3Uvspz+qrNAI9W2X1Rs/53OZlFU7AsNVgASXy+m9M9sitdee42goCAmTZpUZf/atWuJj4/n0qVLzJ49m4sXL6JSqYiPj+eZZ55h/PjxLuO6TMbXvzX1xjeoinqjanbOVXp2j6HDzToOHb2WRCq8Uw9mBmZs0B+60kbEkjYylo0/F/LvReKTikatQBvm/l95f9W9yHL6u84CiVTb/ulLFvp0UROldfyOR4Xb9xWXSTwRg99c244dc+21kZqaSmpqqsdxXaqtcG1LTk7m2LFjdO/evfLn48ePe/zLnCFV71TRusYNb0zayFg27Szk/QXnPXa+qiAqwvlf+043hdCymYac0+4/wPNH3Ysqp+hYUkSqbf/wSTMGk42UpCA01znl6sIUdGkTRG6BlfyiOvBh1Cc/49pwbZOqd6pIXcP6RzNhVBy5l00cyCqh321Vx3UL9WYOZJW4FWvK/Y2I0qk4kmMgr8CMRq2gTXMNfZK1lBmtLFoVGM9g0eUUGQuk6dss1bZfZoRV2408mBrCjLGh7DpqRqWCPp3VqFSwYqv7f/BF6vIY2bXNc6TqnSpKV7vEUADiGml49tEEh+OHj5W4nVi27y+hX4qWviladFol2CCvwMyGnXpWbyoiv9Cz+bci615kOUXGAun6Nku17e/MNFNiKCO1u4ahvTTYbPDbRQuL15s4dcGzh4qB8jOWSo/XXSSRjKXqnSpK13ufnee9z84L0bTzQAk7D7ifhGpCZN2LLKfIWCBd32aptn2AQ79aOPRrmc9xAuVnbKtj5vKyhaaMjEy9pGzzUrfPDR3wkB+VuIckesYytcPcuXPJzMwkMzOTs2fPkpCQwKZNm6o9/+DBg7z77rscPHgQhUJBcnIys2bNokOHDrWoWkbGS+Qx4+qpzmt24+cplRZ77RLDKC2zeO0P+6cp1ywJM/ZdsVsSDo/nYGaRU0tCV/6wUtAlIlZFvHfeeYeQsCjiWnQkOOwqV8vgkx+cl+XCqQOkfzARbWQTbh38JwAO/LSEsQ+msSJ9OY/NcnSO87acT77pfMxw7p9jyDlTzrYDVR8YWSywN9txBd7cP8cIb2Mi28Wkv9Y8NW/c4AgG3BpGaLCSDRklLF5b1bdi0avNhOmquEZk/Ytsr75ik8eMPae+WxKK1OVLrEf+tpGoxi0AWPTGcMqN1ZsKbV7xD1QqNQ/8aSnaqCYA3Jw8lIWvDeWf//wnMNmv5awgr9DKrkzflz5L0ULzRlo1C2Jw73C+WH+VtKHVeyLUpi5f698fbcJt6ljPWBJq67sloUhdvsSqSMQ1UZh3mku/H6Zd8pDKRAygjWpCu+Qh/Pzzz1jNjgsSRNd/BSolBKtrPs8VUrTQvB6FAh4dFcXhE0b2HjV4FcNfVpW+1L+/2oRb1Kd5xnUNqVoSitRVG2W8+Lv91rZZ62SHY81adyMzYwXlhtMEa7v6XVv3JA09O2lQKRXoS6zsyzax8qcyDMb69dx5yO3hxMeq+GC5+4b5tYGv9R/I72Rdm01Rr5KxVC0JReqqjTKWFNlvG7VRjr0WbaS9p2w1F/pd26nzZvZlm8gtsBAarKBzGzUDeoTQrkUQby3WYyz3pFTSpXGUijGpWlZuKSa/0ELjKGkkERH1H9DvZB0bpnCZjM+fP098fHxtafEZdy0Ji821a0wuUldtlLHcZJ9bqgrSOBxTqe37bDbHW2LR2t5cVLU3lXHExLk8C/f2CyM1JYR1O727nZcaj4yMJLfAwvc7xM0fF4GI+g/kd9JWx5KxS7VTp06t/L8rt3upIFVLQpG6aqOMao199ZvF7JhwLeX2fQqFY6KuDW0/7DJQbrbRpa2Pg8gS4fZbQunUVsPC1UVV7Cqliqf1H9DvZB0bM3aZjK9fD3LmzBm/i/GV/CsmInVq1EGOlRtoO05RumqjjOGR9uGJ4kLHp9zFRXbzGmVQVEC0Wa1QVGz1yJ1OqgSpIG1IBIdyjBQWW4mLUREXo6LRH8MUYSEK4mJUhIVII1mA5/UfyO+kTaF0e5MCLlW4stCUItk5V1GpFHS4ueq0oApLwmMniuu8rtooY9OWXQC48Nt+h2MXfjuAQqFAHdIqINqCVBAdoURfUge6kTWgUSvQaVV0Swrh7RlxldsLjzUCoE+3MN6eEUe/HmEBVnoNT+s/oN9Jpcr9TQJ4baHZvXv32tLoNlK1JBSpqzbKGBXbiiYtO5Oz//vKnjDYe8U5+7+nV69eKIMc3ygiUlt4Nb3BUX1DUakUHDpR95/eGU02Plhe4LAtWG2fNnjwuIEPlhewP7v2x8ZF1X8gv5M2hcLtTQoE3EIT6r8loUhdvsQ6unslVwvs5jtlxVewmMvZ9cNHAEREx9Pxtnsrz+0/5gW++mASX773EN36TgDsK/BsNht//vOfefw5xy+RyHLe0yeUNvFBHDtdzhW9lWCNgs5t1bRvpebkOTOb93mWoKRooWmxwp5Mx3JUzKbIvWJxetzfukBc/Yv+TnqERIYf3EUSU9sagiWhSF3exsrMWMHZE7ur7Pt57fsANL/ptirJOL5Nd+5/ejE/r33Pfo4C4hO7M+zR92nfvj3gvEcjqpzHfy+nWSMVvboEow1VYLVCboGFlVtL2bDbgKcP36VqoSkSkbpE1r/Itu8Jtrrx2tRKJJGMG4IloUhd3sYa+/Rij86PT0zm/mkLPbpGVDkP5pRzMEfcUIRULTSdkV9occvH4kZE6hJZ/yLbvidI5cGcu0giGcvIyMgIp44lY9nPWEZGpl5y5dA2t8+N6Xqn2+fOnj2bLVu2cOHCBcLCwujbty/PPfcc0dHRleccPXqUV155haysLKKjo3n00Ucd3iZ9I5Kw0Bw/qik3tQ7j5sRwmjUJ5mKe0Wt7QynEGjvjVJX98bFq7h8cRWJzDTG6IFQqyC8wsz+rjFWbiyjUOw7Apb+bCMDwKUerxmqiYUDPSJI7hdMsVoNareBiXjnb9+pZtfEyRpPzv61r5nUUUsaKcrpzG61Rw+vTYomLCXJqBwl2S0iRupxZOIJ9Xv/YkQmMGhJP07gQCotMbN6ex/ylvzlddODKwtTbWCLbhQhdFdpEtjGRunzGT7MkVCoVs2fPpl27duj1ep577jmef/55PvnkEwCKi4t5/PHHSUtLY+HChWRlZfHEE08QFxfHkCFDqo0riWGKx8Y1r/RhDQ/3bc6fFGPFRKmI1qnYfaiUy0VmrFZo2UzNwN4R3J4cznNvn0Nf7N68zbv7RDFsQDS7DxazZVcRFgt0SQpj0ug47kzRMfONU5jK3b/ZEVlfNzImNYKIcO9uFUXrevrxa566y785Y/fUHZFAuzbaaj11/R1LZLsQqUt0GxNZ957grzHjZ599tvL/jRo1YuLEicycObNy3/r161EqlTz11FMolUq6devG2LFjWbZsmfSTsVS9ZkXFOpJj4EiOoxn70V8NzHy4Cf1vi2D1JkdLSmfs2KcnfV0+pWXXvqTrthZwPtfEuGGxDLojijWb3Xf+agjevFLxk74Rke1CpC6RbSyQfsaezKbQ6/Xo9Y53bjqdDp3OdfvduXPnHzOM7GRnZ9OxY0eUymtttnPnzqSnp7uMI4kRbql6zfrLH7aC/AIzANpQ9z+GE6cNVb4kFWzbY29IrRKCPdLQELx5peIn7S7etAuRukS2sUD6GXuyHHrhwoXcddddDtvCha5nE3333Xekp6fzwgsvVO4rLi4mIqLqFEOdTkdxsevVhpLoGTcU1EEKQoIVqIMUNG+qYcJw+4D/L1nVv3HDXRpH241bCpyMM9Y2UvPmlbqftIh2URu+wd60sYB6jHswZjx58mRGjx7tsN9Vr3jt2rX8/e9/5+OPP6ZTp06V+7VaLZcvX65yrl6vR6vVutTgMhlv3LiRS5cu8dBD9jenjh07litX7BPHn3vuOZfjHzKO3NVLy2P3Na78OfdyOe8vziX7pNHFVTWjVMC4YY0xm21s3eXeba2/kKI3r9T9pEW0C3/7BnvbxgLpZ2xVuN/23BmOuJ709HRmz57NJ598Qo8ePaoca9++PevWrcNqtVYOVWRmZlYZynCGy/ug+fPnk5qaWvmzyWTiq6++YvHixXz++eduC5exs/twKa98fIG3Pr1E+g8FlJRZ0Wl9T1ZTxjWlw01hLF2dy7lL/h1aqQkpevO666lb27EqENEu/KHrerxtY/7W5Qp/ubYtWrSIt99+m08//dQhEQMMGjQIi8XCxx9/jMlk4tChQ6SnpzN+/HiXcV32jMvLy2nWrFnlzz169CA6Opro6GjKyso8KoAMXCmycKXIfou350gpGQdLePPZBDRqBSt/9K5HO2FULCNSY1i3tYD0dZdrvsCPVHjzvv7pZUl58xqMFqJDHP2XwTs/aVGxKhDRLvyhqwJf2pg/ddWEv5ZDv/baawQFBTnMG167di3x8fFotVrmz5/Pyy+/zNy5c4mOjmbq1KkMHTrUZVyXyfjGp4t/+9vfKv9fMVwh4z2/Xyjn1Fkjg/vovErGaSNiGTc8lg3bC/hwiefLZ0XizJsXIFpX1Zu3uNRKqaF21xnlXzHRukU46iCFw+2yN37SomJVhzftwl+6fG1jtVFf1eGvqW3HjtW87Lxjx4588cUXHsV1qbZr1658+eWXDvuXL19O165dnVwh4ykatcIro/S0EbGkjYxl48+F/HtRYBMxSNubt675SVfE86Rd+EOXiDYWSD/jemWh+Ze//IWpU6fy7bffVj4tzMzMxGQy8eGHH9aKwPpAVISKwquOT6A73RRCy2YaMk94Nv1r3PDGpI2MZdPOQt5fcN6vzlfuUuHNeyMRYUoeHhnJweMGfvqljDMXa9+H+MdteUwc25IHRiVw6Oi1nqa3ftKiYolsFyJ1gbg2JlqXJ3jyAE8KuEzGjRo1Yvny5ezcuZMTJ04A0K9fP3r37i1UhBS9ZkXGmnJ/I6J0Ko7kGMgrMKNRK2jTXEOfZC1lRiuLVrmvaVj/aCaMiiP3sokDWSX0u62qyXuh3syBLPcfnjUEb16p+EnfiMh2IVKXyDYWSD/jemmh2bt3b+EJ+Hqk6jUrKtb2/SX0S9HSN0WLTqsEG+QVmNmwU8/qTUXkF7o/b7Ndov1loXGNNDz7aILD8cPHSjxKxg3Bmxek4Sd9IyLbhUhdottYwPyM65hrmyQWfUjVa1ZUrJ0HSth5QMxUr/c+O897n50XEgsahjcvSMNP+kZEtgsQp0t0GwuYn3Ed6xnLFpoyMjL1klO/nnD73MS2N/lRiXtIomcsIyMjI5q61jOu1WRcndfsn6Zcs9jL2HfFbrE3PJ6DmUVOLfZcec16G8uZp+7Gz1MqbRzbJYZRWmZxy8/4Rn/YyWPiKi0Js0+WVloS9r01klNnDE4tCdfM6yhMV8U1N/rpdm4Xwn13R3H8N2MVC8cBt0VQarBWa+GY/m6iUF35mzMo3HUAw7lLWM1mdJ1vpvnkMZj1JWxLGYUp79r4cLsXp3LzS0+T+8M28r7fiqW0jNAW8UR0SaLpqIEOZfS2nNX5SfvyWT7zgeM0rugIBdNGhxKsUZBxtJy8AishwQriGynJ/t3C/hxzlfPfm64VpqtCmwhdFdpu1OVrnfmCVRo+aG4T8J6xVO0NQZyNo1RtL0VaOPqia9egyQ77Lm/bS4/l79N80hhO/ms+AI1Se3PzS09z7KX3OfH6Rw7XDCt3Ps4sZQvTCYNCUCrhrWWl6Eu9HzGUqi5/aHMXWx1LxgFXK2V7Q1E2jnXB9vJ6vLFwBLG6yn63f4bq6GuLBW768/9gvJTPr/+cC4AqPMyntzkE2sK0TbyStvEqNv1Sjr7UhlIJai+7R1LVJVqbJ9hQuL1JgYD3jKVub+hPpGJ76U9rT3dRBmtQacNRhWjQdriJ9m/MAiB3nX0oShUWSsydKeSt+4kWj95PuxemEpLQBEtpGZe+3UTms6/V+DukZmHasbX961dw1crjw0Po0EqFSqkgt8DKD3tM7DvmOBRQX3R5q80TpJJk3SXgyVjq9ob+Qkq2l/6y9vSEFo+NpfP717xPSk+dZf+kWRTs2AdA2E0tUQYFEdXzFhrf3Ydf3/oP+kPZxNyRQuL0SUR0Sarxd0jNwjQuyt4jfzA1mLxCG8s2GFGpYECymomDQlApDezO8j7xSVWXL9o8QU7GHuKuxV6xuea/niJj+ZsKS8KFX18KuO3l7sOlnMu9QIhGSWJzDSmdwoRYe3rCpVUbKc4+SZA2DF23jjQZkYqm8bW37QZF2B9cBcc14tCTL3Dmv19VXmfWF3Pz36bX+Dv8VU5vP8vgP8zMjOXw4TdllU53h0+a+eukcIb11rAny4y3XQep6vJFmyfUq2T86quvonAxJvfiiy/6LEDq9ob+QEq2l+Afa09PMZy7hOGc3afg0uofufjNevrs/ApVaAi/vvUfLGX2pdQ2i4VzS1ZVufbs4pVuJWOpWZiW/9G5/OW4uYrlaJkRjpwyc1sHNXHRCi4VeJ72pKrLV22eYLUF/JGYR7hU27lzZzp16kSnTp3YtGlT5f8rNhHkXzERqVOjDnJM+t7YG4qK5S+kZHtZHddbOAaKq4ePoT9wlFb/kwaA4ax9NkR5gR6rqarZkPGCd/4GvpbT18+yqNjeFp3NVqjYFxrsee9OqrpEaPOEevUA7/p3Qi1cuNDpO6J8JTvnKj27x9DhZl0VV6cKi72Dme73WETG8gdSs710hbfWniJRhYagjrGb1JhyL1N6+hyhLZqhDA3BWnbNdCikeVOvf0cgLUxPX7LQp4uaKK1jMogKt+8rLvOs8yBVXaK0eYJUkqy7uN0KXQ1X+MKP2/KwWm08MKqqIYm39oaiYolGiraXURHOx0srLBxzTtfeA7wbadSvJxGd2lG462DlvnNLV6NQKmn1xINVzm31pOvX2Ygup6jP8vBJMwaTjZSkIDTqa/t1YQq6tAkit8BKfpH7waWqS6Q2T7DZFG5vUiDgD/Ckam8I4mwcpWp7KdLC0Rddt2//gsubMyj7/TzKkGAiu3ci/oF7MF8tIev/vVl53sm359Fs9CA6/PP/CG+XaJ9N0acHCWkjyd+0k8apzp0FpWphWmaEVduNPJgawoyxoew6akalgj6d1ahUsGKr+38kpKpLtDZPsNaxnrHLZJycnFzZIzYYDHTv3h0Am82GQqHgl19+ESJCivaGIM7GUaq2l6ItHL3VZcovIOGhUWhiY8Bmo+z0eX6f9wW//utTDGeu3c6ar5bw84CHSHr5TzQZcRctHrkPw9lLnHjjY3Je+4ihxc6XXkvZwnRnppkSQxmp3TUM7aXBZoPfLlpYvN7EqQvuP2yWqi5/aHOXuvYAz2Uy3r9/f62IkKK9IYizcZSq7aVoC0dvde2993/cPrf8cgFHpv2dI9P+7vY1UrYwBTj0q4VDv/r2gl+p6gL/aHOHujZmHPBhChkZGRl/IJWxYHeR/YxlZGTqJXuOFbp97q1JUW6fu3btWpYuXUp2djYlJSUOb4tOSkoiODgYlerag+Ply5eTlOR6lWit9oyd2S4CjB/VlJtah3FzYjjNmgRzMc/ollWlMwtNhQLGjkxg1JB4msaFUFhkYvP2POYv/c3pgg9XdpzexnJWTm/LKCJWRTxnVokAYcEwMEVDlzZBRGkVGMvhwmUL63aZOHnesZzvTdcK1eXM9jI+Vs39g6NIbK4hRheESmU39tmfVcaqzUUUOvEzSH83UWgsaBjtQkQZK8opMpav+KtnrNPpSEtLw2Aw8MILLzg9Z968efTs2dOjuJIYpnhsXPNKH9zwcN+Wpz79+DU/4+XfnLH7GY9IoF0brVM/49qKJbKMImO58q2NDFcC7j+sEakrJkpFtE7F7kOlVTyIB/aO4Pbk8Gq9lv0dCxpGu5Dq98gTPHnMqNfr0ev1Dvt1Oh06XdVFQXfeeScAu3bt8kWeA5JIxqL8eRuCN7LoWCJ9a0XqEulBLDJWQ2gXUv4eeYInsykWLlzAnDlzHPZPmzaN6dNrXmp/IzNnzqS8vJz4+HjGjx/PAw88UOM1kpj7IcoHtyF4I4uMJdq31t8+y+C917KoWA2hXUj5e+QJniz6mDx5Mj/++KPDNnmy44sPamLBggVs3LiRbdu2MWPGDN5++22WLVtW43WS6BmLoiF7I3tDbfnW+oJID2IRsRpCu6gv3yNPprY5G47wlt69ry0+6tu3Lw8//DCrV68mLS3N5XX1Khk3VG9kb6kN31pfEelBLCJWQ2gX9eV7ZJXIR6BUKnFn0lq9SsYN1RvZW2rDt9ZXRHoQi4jVENpFffke+WvRh8ViwWw2U15udw80Gu1/zNVqNVlZWVitVpKSklAqlezatYvPPvuMqVOn1hjX7eXQ1yN6ObQoGqI3si/427dWBCI9iEXEagjtor58j/w1tW3VqlU8//zzlT937doVgEWLFlFSUsLs2bO5ePEiKpWK+Ph4nnnmGcaPd21mBRJZDi2K/CsmWrcIRx2kcLgt8sYbWVQsqeK+b610ynm9B7GvxvfexGoI7aK+fI8sfkrGY8aMYcyYMdUeT01N9SquJGZTiCI75yoqlYION1cdiK/wMz52wvmiB3/HkiqnL9l7iaJ9a/2NSK9lT2M1hHZRX75Hdc1Cs14l44bijSwKf/jWikKkB7HIWA2hXdSX75HN5v4mBSTxAE+UP29D8EYWGUu0b63IMor0IBYZqyG0Cyl/jzxBdm3zAlH+vFD/vZFFxxLpWytSl0gPYtG+zQ2hXUj1e+QJUpna5i6SSMai/Hmh/nsji44F4nxrReoS6UEs2re5IbQLqX6PPPu9datnLFtoysjI1EvW/OL+gqXh3QPfLw28AhkZGRk/UNe6mbWajJ15nQL8aco1i72MfVfsFnvD4zmYWeTUYs+VP+zGz1MqbQTbJYZRWmbx2h/W21jOyhnIMlZcIzKWyM9SqroAXv3csXc1+S4VzRvDod9snM23oQ6Czi2VJDRW8PNRKz8erDrW/tfx9q/ZXz6t+kC0abSCpjGOt9JNYxT07RpE5m8Wlv5Y9fe//pj9AZ2INlZRTpGxRH6WviKVKWvuEvCesWxJWD/KCGLLKVVdCY2gZZyCjGwrG/ZfS7p7cyw8NUxF95sU/HjQPV0XC2xcdLK68d6m9q/l3uPuP0CVsu2lyM/SE+raA7yAzzOWLQnrRxlBbDmlqitY7XwxjNUKpUYw+eirpA6Crm2UFBbbOH7W/WQsZdvL2rBWdYY8z9hDZEvC+lFGkG45Reo6f9lGmclG7w5KCkusnLtsQ62CW9ooaRYN3+31zWehS6KSEI2CnzPNHiWJ+mJ7KRJ/LYf2F9Um4+pMggA0Gg0tW7ZkxowZVbw7vUG2JKwfZQTpllOkLkM5fPGThRG3qbj/jmsr+4zlNr7abuXYOd/KlnKzCqvNxr7jns19ri+2lyKRSo/XXapNxq5MgiwWCzk5OcyaNYs1a9b4JEC2JKwfZQTpllO0rnIz5BbZOHbO/gAvVAMp7ZSMvl3JF9usnLroXRZoHKmgdVMlJ85ZKfDQsqG+2F6KpK4lY6/GjFUqFe3bt2fChAk+CzAYLaiDnMuoT5aE9b2MIN1yitQVFwkPD1Rx8qKNHw9YOXbWxoGTNhZstFBsgOG3KqnmhrJGUm62a9nrYa8YxJZRqp+jp1htCrc3KeDTA7xx48b5LCD/iolInRp1kGOF1CdLwvpeRpBuOUXq6tleiTpIQdbvVc83WyDnvI0orYKocM81KhWQfJOKEoONzN88T3QiyyjVz9FT6toDvIDPppAtCetHGUG65RSpKyLU/q+z3q9SUfVfT2jfUklEmIIDJyxVjP7dpb7YXorEYnV/kwIBT8ayJWH9KCNIt5wideX94UF/S5uqGTdYDUkJCsqMNq54kauuDVF4lxnqi+2lSOqan3HAp7bJloT1o4wgtpxS1bX7mJWuiQruukVJXKSNM388wEtua+/ZfrfH4vFtb0QYtGuu5Eyu1etXXEnZ9lLkZ+kJUhl+cJeAJ2OQLQnrSxlBXDmlqquoFP673sKdnZUkNlHQqZUCswUuFsDG/Rayz3qeAbq3U6FSKthz3LcVI1K1vRT9WbpLXVuBJ4lkLFsSeoZUywjiyilVXQAFxbA6Q9xA49aDFrYe9H2amFRtL0V/lu5S13rGAR8zlpGRkfEH/ppNsXbtWtLS0ujevTtJSUkOx48ePcq4ceO45ZZb6N+/P4sWLXIrruxnLCMjUy+Zt9H9c6cMdP/cbdu2UVRUhMFg4IUXXuDYsWs9/+LiYgYNGkRaWhpTpkwhKyuLJ554gldeeYUhQ4a4jCsJC02FAsaOHu6UyQAAG3hJREFUTGDUkHiaxoVQWGRi8/Y85i/9zenkclf2kuNHNeWm1mHcnBhOsybBXMwzem17WV90VWgbPuWow/74JhoG9IwkuVM4zWI1qNUKLuaVs32vnlUbL2M0Of6tXjOvo5AyVpTzmQ+qn34QFgwDUzR0aRNElFaBsRwuXLawbpeJk+erlvW96VqnurzRVlH/Y2ecqrI/PlbN/YOjSGyuIUYXhEoF+QVm9meVsWpzEYV6x+GGt56G1atX89PXyzlbVIzJbKV5VDgDb27OQ93bEaq+9jXs/s5X1WoCeOaZZ/jf//1fYfU/tv8+du7ax8kTx7l06QKxcU34z2fLHc7LvXSRJx8d7zLW7Nmz+ffn8UJ0ibDQtHowkqTX69Hr9Q77dTodOl3VKX533nknALt27XI4f/369SiVSp566imUSiXdunVj7NixLFu2TFrJuDqefvyad+ryb87YvVNHJNCujbZa79TqeGxc80rv1PBw528FlnVd4+4+UQwbEM3ug8Vs2VWExQJdksKYNDqOO1N0zHzjFKZy9wKKLCNAdISCaaNDCdYoyDhaTl6BlZBgBfGNlESGKwH3v22itMVEqYjWqdh9qJTLRWasVmjZTM3A3hHcnhzOc2+fQ19cVdeKFStYunQp/Vo24p4OLQlSKthzJo+PdmSy4dhZFo5PJURt1/TqkFud/t65O49ytqiEAQMGCC3jO++8gzZCR9u27Sgpqf6Poi4ykj/N/IvTY/M+eR+T0cQdd9zBvz8/KUSXCDz5HixcuJA5c+Y47J82bRrTp093O052djYdO3ZEqbw2Aty5c2fS09NrvDbgyViq3qkNRdeOfXrS1+VTWnYtgazbWsD5XBPjhsUy6I4o1mwucCuWaN/aCYNCUCrhrWWl6Et9G00Tpe1IjoEjORcd9h/91cDMh5vQ/7YIVm8qqnJs8ODBPPnkkyiXvVm57/5b2tJyxxE+3ZXNyiOnGJd8EwDDOrZyiH3paikv/VBCxybRtG/fXmgZN27cyFWjPUE+/dQjGMqcvwsxJCSU/ql3O+zPzsqktKSE3n36ERMTAzgm40D5GXuSjCdPnszo0aMd9t/YK66J4uJiIiKqzhzR6XQUF9c8+TzgD/Ck6p3aUHSdOG2okogr2LbHfsvWKiHY7VgifWvbxCtpG69i0y/l6EttKJV2r19v8benbn6BfVqaNtTxK9WlSxeHLyjAoJtbAPDrZcfb4+tZnfkbVhvc2yXR5XnelLFFixYeX3M9G39YC8Ddg++p9pxA+Rlbbe5vOp2O5s2bO2yeJmOtVuuQePV6PVqttsZrA94zlqp3akPX1ThaDUCBkzHQ2qBja3vTLLhq5fHhIXRoZZ+Lm1tg5Yc9JvYd89HF3UfUQQpCghWogxQ0b6phwvBoAH7JKnU7Rm6x/dyYsOr/4NlsNlZnniZUrWJIkm+JUzRlZWXs2L6F2Lgm3JLs+xivaDybmyBmFV779u1Zt24dVqu1cqgiMzPT5R1NBS57xhcuXKj22ObNmz2U6ZyavFOjIzUEOTEs8TcNWZdSAeOGNcZstrF1V1HNF/iBuCh703wwNZiwYAXLNhhZttGAxWpj4qAQbusQ2H7EXb20/PcfrZj795b89X+aEh6q5P3FuWSfNNZ8MWCx2piXkUWQUsHQ9i2rPW/3mVzOFZUw8ObmaIPVouQLYcdPmzCUlXHX3UOrjJFKBYvF/c2zuBaMRiPl5eUAGI1GjEYjVquVQYMGYbFY+PjjjzGZTBw6dIj09HTGj3f98BNqSMaPPPIIZ886Tvr+6quveO211zwrQTW4651a2zRkXVPGNaXDTWEsXZ3LuUuBucUM1tj/NZbDh9+Use+4md1ZZv69ooxSg41hvTWC+jLesftwKa98fIG3Pr1E+g8FlJRZ0Wndfzj19pYDHLpwhf+5vROtY6q/m1l5+DcA7u3seogiEGxY/x1KpZLUga5nCQQKf80zXrVqFV27duWxxx4DoGvXrnTt2pU9e/ag1WqZP38+P/30EykpKUyfPp2pU6cydOjQGuO67F48//zzPPbYY8ydO5fWrVsDMHfuXNasWcOSJUs8K0E1GIwWokM0To8F2gO3IeqaMCqWEakxrNtaQPq6y17H8ZXyP0YhfjluruKqVWaEI6fM3NZBTVy0wmsvB1+5UmThSpG9S7XnSCkZB0t489kENGoFK390fTfx0Y4jfHHgV8Z0SeTR26q/fS0qM7H5xDlax0SQnNBYqH5fOfP7bxzPPkpy91uJjWsSaDlO8ddy6DFjxjBmzJhqj3fs2JEvvvjC47guk3G/fv3QaDRMmTKFDz/8kPT0dA4fPsySJUuIjIz0+Jc5I/+KidYtwlEHKRxuvQPtgdvQdKWNiGXc8Fg2bC/gwyXVD1HVBkXF9jI4m0VRsS80WAFIY83S7xfKOXXWyOA+OpfJ+JOfM5m/K5uRnVrzwsDuLmOuy/4dk8XKvZ1bC1brOxvXfwfAwMHDAqykeuracrYa72d79+7NG2+8wcSJEzlz5gwLFy4UlohBut6pDU1X2ohY0kbGsvHnQv69KLCJGOD0JXuvM0rrOBgRFe78Dc2BRqNWoA2r/iv1yc+Z/CcjixEdW/G3QT2qfcdkBSuPnCJIqWC4k+lugaS8vJwtmzagi4zitl59Ai2nWmxWm9ubFHCZjJOTk+nevTtTpkyhpKSEjIwMevfuXblfBFL1Tm1IusYNb0zayFg27Szk/QXnJdGjOHzSjMFkIyUpCM11z610YQq6tAkit8BKflHtC42KcD4u3OmmEFo205Bz2vkDvDlz5vCfjCyGdWjJS4NTUNaQiI9evMLxvCL6toknJizEZ90i2bPrZ/RFhfRPvZugoIBPyKqWumYu77ImXb2UVBRS9U5tKLqG9Y9mwqg4ci+bOJBVQr/bqt71FOrNHMgqcSuWSN/aMiOs2m7kwdQQZowNZddRMyoV9OmsRqWCFVvdm7UgWtuU+xsRpVNxJMdAXoEZjVpBm+Ya+iRrKTNaWbTKMc7SpUv54IMPaBoRRs9WTViX9XuV443CQ+jVquq468ojvwEwuktrv5Zx5cqVHDxi91bQFxVhLi8nffliAGLjmtA/dZDDNZVDFIPcG6IIlJ+xVSI9XneRxJ81qXqnNgRd7RLt7xGKa6Th2UcTHI4fPlbidjIW7Vu7M9NMiaGM1O4ahvbSYLPBbxctLF5v4tQFz7ozorRt319CvxQtfVO06LRKsEFegZkNO/Ws3lREfqHjPKnDh+0+DBevlvK37/c4HO/RvHGVZGwot/D9sTM0jQild+umfi3jihUr2L17d5V9yxb/F4BOXW5xSMb5ebkc3L+X9h060aKle8MngfIzlsIdnidIIhlL1Tu1Ieh677PzvPfZeQGq/ONbe+hXC4d+db5E1xNEadt5oISdB9z741TBm2++yZtvvknJ3BfcOj9EreKnqaM81uZNGRcvXszRE+5//o1j41jx7Y8e/Q7Zz9g9ZAtNGRmZesmrn7u/SvOv4wPfLw28AhkZGRk/YJPIgzl3kYSf8Z+mXLOEzNh3xW4JOTyeg5lFTi0hK3yDncXzNpYzf9iNn6dUWv+1SwyjtMzitQext7pu9NPt3C6E++6O4vhvxioWjgNui6DUYHVq4QiQ/m6iUz/jyWPiKi00s0+WVlpo9r01klNnDE4tNKvzM/a0viquEamrobQxke1i0l9rnso4bnAEA24NIzRYyYaMEhavdTQ3WvRqMyH1BdfqzBcslrp10x/wnrFIS8iGYHvpjYWjK6RqoSlSV0NoY6LbxfW0ahbE4N7hfLH+KmlDPXMxE11fnlDXRmAD7u4h0hKyodheOsOVhaMrpGqhKVJXQ2hj1eFtu6hAoYBHR0Vx+ISRvUcNHl9fG22/Ojyx0JQCAe8Zi7SEbEi2lyIsHF0RaAvN6vBGV0NoYxWIbhdDbg8nPlbFB8vduwu5kUDWl1RW1rlLwJNxTZaQXTtGEhSkcMtvQWQskfhD1129tDx23zXzmNzL5R5ZOLpCChaazvBWV0NoYxWIbBeNo1SMSdWycksx+YUWGkd5/sqkQNZXHRul8D4ZL1iwgIcffthnAe5aQhaba+4JiYwlEn/o2n24lHO5FwjRKElsriGlU5hHFo6uqLDQXPj1pYBZaDrDW10NoY1VILJdPDIyktwCC9/v8Gxe9fUEsr4azAo8UclYpCVkQ7K99MXC0RVSsdC8EV90NYQ2VoGodnH7LaF0aqvh9U8v++TdEMj6stax2RReP8AT9aQy/4qJSJ0atZO3U3hqCSkylkhqQ9f1Fo7eIiULzevxVVdDaGPV4U27CFJB2pAIDuUYKSy2EhejIi5GRaM/hinCQhTExagIC6nZ3j+Q9WW12dzepIDXybgm+z93EWkJ2dBsL2+kJgtHV0jNQrMCEboaQhtzhaftQqNWoNOq6JYUwtsz4iq3Fx5rBECfbmG8PSOOfj3CaowVyPqy2Wxub1LA5TBFcnKy06Rrs9kwGn1/UAR2S8iJY1vywKgEDh29dhvljSWkyFgiEakrKkJF4VXH8bUKC8fME55PP5KihSaI09UQ2pjIdmE02ZzOnogIU/LwyEgOHjfw0y9lnLlYXmOsQNZXvRozrmsWmg3B9tIbC0dXSNVCU6SuhtDGRLYLixX2ZDom74rZFLlXLE6PO0N0fXmCVDoV7hLwqW0g1hKyvtteemPh6AqpWmiK1AX1v42JbhciEVlfnmCRimu8m0giGYu0hKzvtpfeWDi6QqoWmiJ1Qf1vY6LbhTPyCy1u+VjciMj68gR/Lfr44IMP+OijjwgJufYGlgEDBvDOO+/4FFcSyVhGRkZGNP5cgZeSksLixYuFxpT9jGVkZOol/zu70O1zP34uyu1zP/jgA3bv3i08GUvCQlOhgLEjExg1JJ6mcSEUFpnYvD2P+Ut/czoh3JW9obexnNkbjh/VlJtah3FzYjjNmgRzMc/olb1hfKya+wdHkdhcQ4wuCJXKbuCyP6uMVZuLKHTis5D+biL8//bOPaipK4/j3yQkEPMoD5ECsjwULBa0vCqt1bYKtArWoduCrUVWS1u7grU72pnd7u50mbErq0VH3KUvhzqidSuoIGoF7NSxuMUHUsVqxQq1LSIPwQQJIZDsH6mxMZeYXE7IgZ7PzJ1hDuSXX85Jfpx77rmfABZ6ST8fCZ6ccR+iHpTB11sCsViA1nYdvjqlQll1J7T93P9bKz6ayqmq5BPPmqqST/+TzItrHAH+Y0nyfbGqgHsb1zhXICFWgsgQF7jLBdDqgGudgzhU248rLeZ9tilHDoDMex8AMlIu4lTtcTRevoJr19vgM8Ebn24t5Pxbg8GA/Z9XouLzKlz9qQVisQumTglD5gtpmPpAGPzDIonlRUKhac/MWKVSQaWy1IIqlUoolZZ7tBsaGhAfHw+pVIro6GisWrUKAQEBw8qXimWKlVl3fKe79v5o9J0u8EdoiHxI3+lIxHp50USTa1Ym43+rsae7CB5KEU6c7TVzzSY8osCjUbIhXbNcJM50N3l+v6y9afL8LkmdgFmxSk7P70jGI9X/pPMiNZakY3koBMhOlcJVIsDX3+rQ3qWHm6sAfl5C3CcTArD9IhSfvs/Pz4dSIUfopBD03LIuE9pU+BH2H6rEQ5EP4tU/vAStVouKw9V4889/R17uX+EfFkksLxLYc9K/bds2bNmyxaI9OzsbOTk5Zm1PPfUUnn32Wfj5+aGtrQ3vvfceli5dirKyMshkMt75Or0YM9esfa5Zkp5f0vFI9j/p10nStUwy1ktJbhAKgX/t7IWql39V4tv31dXVEGqMp/PLVrwJTR/3lrXLV5qw/1AlHo5+CP98523T/QcLnk5C5utvIH/LB5iXmk4sLxLYs5siMzMTqampFu1cs+KwsDDTzz4+Pli7di1iY2Nx5swZPPbYY/ySBfMZW4VG1yxJzy/peCT7n/TrJDmWpGKF+AkxyU+EL+p0UPUaIBQCYp7TI759b+up9ZmzDQCApLlPmN0IJpfLMHNGHH5quYa6ujpieZHAoDfYfCiVSkycONHi4CrGdyMQCCAQCIZ9J5/TZ8bMNUvGQUzaP+xsbzDJvGhlapDx49el1iMrxQ3hgSKIhAK0delx+GQ/Tn9n+xdqOrrvdTpjLq6ulv8EXV2NIqD6+noAoSOalzUctZvi4MGDiI+Ph6enJzo7O7FhwwZ4enoiKipqWHGdXoyZa3b4DmLS/mEavMEk86KVCe7GM6L0Oa5o7zZgZ5UWIhHwZJQYGUluEAn7cOKCbQXZ0X0fFGicQZ8524CZM+JM7QaDAd80GJcfWltbcXcxduZn0lECoPLycuTm5kKj0UCpVCIuLg5FRUWQy+XDimu1GC9fvtzqg99///1hPTnAXLMkHMSk/cM0eINJ5kUrv0woodUB/96rMakqz10ZwN+WyJD8iAQnLwzAlpLi6L6fEROFwICJKD94GOM9PTDrkRno02pRsq8CzVd/BABoNJoRz8sajpoZk6h7XFgtxvX19fD19UVycjKmT5/uELsRc80Oz0FM2j9MizeYZF608suZP+ouDZg5gzVaoKFpAA+HizHBQ4DrXff+3Dn6vS8SibDunbeRt2kLPvykGB9+UgwACAkKxCuZi1G4dRvnzNCZn8nRdguF1WJcU1ODmpoaHDhwABUVFXj88ceRkpKC0NBQaw+zi44b/QgKkEHsIrA4leHjmiUVayT4tWuWTzEm7R8m4Q12RP/T6lkeLjd7jH3BtYvidpvUVQDYMDceife+zwRv5L/7D1xva0drWzuUCgWCAwNQduBzAEBISAhwfOTzGorBIWbktGL1Mr5IJMLs2bORl5eHzz77DIGBgcjIyEBxcTGxBJhrlp+DmLR/mDZvMMm8aOWH68azJHe5pabWXWZs69HYVqhG8r3vM8Eb0yOmIviXdeTaU3UQCoWc27qYz9h27lkF+vv7UVlZidWrV2PHjh3IyMhAYmIisQSOHGuHXm9A2kJzMxdf1yypWCRxV3CvC992zTb+YN8FPNL+YZLeYJL9T6tnmRTnrgygr9+A2CkukIjvtCvHCRAZ4oK2Lj06btr2op313q+pPYmvT9Uh8YnZ8Pe3tOs58zNp0OttPmjA6jLFW2+9hcbGRsyePRvZ2dlmm51JwVyz9rlmSXp+Sccj2f+kXydJ1zKpWBotUPaVFulz3PDm81LUfjsAkQiYGSGGSASUHrX9nzTfvt+3bx8unjN6y2+qVNDpBrD9vyUAAB9vbyTNuXNb8vrN/4HBYMDk4CBIXCVo+PYiqr88himhk7Hi1aVE8yLBmJLLl5eXQyqVorm52UyKYTAYIBAIODd584G5Zm2/kkza80urN5h0XiRdyyRj/e/8AG71aTAnWoJ58RIYDEBz6yC2V/aj6Zp9MzY+fV9aWooTJ06YtRUV7wIATI+YalaMHwidjIrD1Th2vBa6gQH4+96PpYvT8dzCFM79x8PJiwS0LD/YitVifPHixRFJgrlmbYe055dWbzDpvEi6lknGAoCz3w/i7PeW28LshU/fb9++HT9fGlpy9GsWzEvCgnlJI5IXCfSj7AKe02/6YDAYDEegN4yuYsx8xgwGw6F0dHRg8+bNOHr0KDo7OzF+/HgkJCRg5cqVNrkf+JKa3Wjz3+7dQm67Ll9GdGY8lGu2+tNYk5IwNHgcejWDNvlhudypb7xyR9f39ekbRl1fih++OX+TU9dnzY1MQyyuPrO3v24/htZYd/ufASAi1A2/T3THpWatmXb0yYcV6O3Tc2pHd28MHtKzPFbG0ppnmYaxzC8zf/Gank7sKXgevap2hM9IQ+ijobjR2ogdO3fh4JFTWPjHnRBLpBax/rTQcrufvTjymz4cARXLFKSUhLTqOGlVe9Ici6R2lI2l82LVffEBerpaMPeFDZgclWJqvz8wCkc+XY1zxz5B9NzXeedpjdF20u90hSZATklIq46TZrUnrbGGgo92lI2l82K1fF8LF7EbJj2UbNY+afp8iFxc8d3JPUSehwu9Xm/zQQNUzIxJQauOk3a1J82Q0I6ysXQe+oF+iFxczRzIACAQCuEidoPqxo/Q3OqCVOZB/rkHR5dmdUwVY1p1nLSrPWmGhHaUjaXz8PCZjO72KnS0XMB4v3BTe0fLBWg1xmWmnq4WhxRjtmbsRGjVcdKu9qQZEtpRNpbOI3JWJprPH0F18So8uuAv8Lg/FF3XL+N4+bsQisTQD+owoOP+qqfhMqaKMdcX9N1GIBBgxYoVxBMaDrTqOGlXe9IMCe0oG0vn4Rsci7mL83G8bC0OFb0GABAIRXgg7jlobt1Ac0MVJG7Dk7IPxWjbZ2y1GI8bN86iTaPRoKSkBN3d3dQVY1p1nKNN7UkzfLSjbCydy6RpTyM4IhE3Wi9Bp70Fd+9gSOVe2FPwPIRCFyi9fueQ5x1tM2Orl6SXLVtmOtLS0tDX14fS0lLMnz8f1dXVI5WjzdCq4xyNak+asVc7ysbS+QiFIoz3C4dvcCykci/0qtvR+fMF+IbEce4zJsFos7bd8x3d3d2NjRs34plnnsHg4CD27t2LNWvWwMvLayTyswtadZy0qj1phqR2lI0lXRj0etSUrYXeMIioOa857Hn0g4M2HzRgdZkiLy8PVVVVSEtLw/79+yGTyRySBCklIa06TlrVnjTHIqkdZWPpvFg67S3sKUhDcEQCFB4T0d+nxuX6A+j4+Tzinl4F/8nxNudkL2NKoVlUVASJRILCwkKzL+EjrdAkqSSkVcdJo9qT5lgktaMAG0tnxRKKxPDynYLLZyrQq26Hi1gK74AIzH/5IwRMmWVzPnxw5PKDXq/Hpk2bUFJSAo1Gg+joaOTm5nIK9m2FCoUmSSUhrTpOGtWeNMciqR0F2Fg6K5bIRYKExflEYtmLIy/gffzxx6ioqEBxcTF8fHywbt06LF++HGVlZRAK+d3YTMXt0AwGg0Eag0Fv82Evu3btQlZWFkJCQiCTybBmzRo0NTXh9OnTvPNlCk0GgzEmGcrgx8XBHVFQqVQW7Uql0kLzqVarERsbi927d2PatGmm9uTkZKSnp2PJkiW88h1Td+AxGAzGbW6rS22hoKCA8ya37Oxs5OTkmLX19Bi3L95dpBUKhel3fGDFmMFg/ObJzMxEamqqRTuX/F4uN94xqFarzdrVarXpd3xgxZjBYPzm4VqOGAqFQgF/f380NDQgMjISgLEQX716FeHh4fd49NCwC3gMBoNhJ4sWLcLWrVvR1NSE3t5erF+/HkFBQYiJieEdk82MGQwGw06ysrKgVqvx4osvQqPRICYmBoWFhby3tQFsNwWDwWBQAVumYDAYDApgxZjBYDAogBVjBoPBoABWjBkMBoMCWDFmMBgMCmDFmMFgMCiAFWMGg8GgAFaMGQwGgwJYMWYwGAwK+D8nJZ1wwcBj/gAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aw0fgUrUIQAy"
      },
      "source": [
        "## **Model 6** : LSTM (fixed-length inputs + word2vec word-embeddings + frozen word-embedding weights)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B6r1ME9SJqNS",
        "outputId": "3b46af5e-8088-4ace-8c86-07307a4228ae"
      },
      "source": [
        "w2vec_fixeds_freeze = LSTM_w2vec_fixed_size(vocab_size,\n",
        "                         embedding_dim = w2vecembedding.vector_size,\n",
        "                         hidden_dim = 50,\n",
        "                         vec_weights = pretrained_weights,\n",
        "                         freeze_emd = True)\n",
        "train_model(w2vec_fixeds_freeze, epochs=200, lr=0.01)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train loss 2.603, val loss 2.556, val accuracy 0.160\n",
            "train loss 2.322, val loss 2.457, val accuracy 0.179\n",
            "train loss 2.105, val loss 2.505, val accuracy 0.179\n",
            "train loss 1.934, val loss 2.475, val accuracy 0.184\n",
            "train loss 1.658, val loss 2.537, val accuracy 0.203\n",
            "train loss 1.566, val loss 2.472, val accuracy 0.227\n",
            "train loss 2.028, val loss 2.638, val accuracy 0.173\n",
            "train loss 2.052, val loss 2.512, val accuracy 0.221\n",
            "train loss 1.885, val loss 2.543, val accuracy 0.157\n",
            "train loss 1.730, val loss 2.565, val accuracy 0.179\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 664
        },
        "id": "-_231r36Op_W",
        "outputId": "ac67b7b7-319e-4f23-d364-afd979e2898b"
      },
      "source": [
        "print(\"***Model 6 : LSTM (fixed-length inputs + word2vec word-embeddings + frozen word-embedding weights)***\")\n",
        "pred_vs_true_m6 = matrix_pred(w2vec_fixeds_freeze, val_dl)\n",
        "diagnostics(w2vec_fixeds_freeze, val_dl)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "***Model 6 : LSTM (fixed-length inputs + word2vec word-embeddings + frozen word-embedding weights)***\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           A       0.00      0.00      0.00        22\n",
            "           B       0.00      0.00      0.00        12\n",
            "           C       0.00      0.00      0.00        13\n",
            "           D       0.11      0.05      0.06        22\n",
            "           E       0.38      0.15      0.21        20\n",
            "           F       0.40      0.07      0.12        29\n",
            "           G       0.00      0.00      0.00        19\n",
            "           H       0.18      0.72      0.29        65\n",
            "           I       0.17      0.06      0.08        18\n",
            "           J       0.38      0.43      0.40        21\n",
            "           K       0.20      0.08      0.12        61\n",
            "           L       0.50      0.12      0.19        17\n",
            "           M       0.17      0.05      0.08        37\n",
            "           N       0.00      0.00      0.00        19\n",
            "\n",
            "    accuracy                           0.19       375\n",
            "   macro avg       0.18      0.12      0.11       375\n",
            "weighted avg       0.19      0.19      0.14       375\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEDCAYAAADgNmW7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd3hUVf6H35lJLxMSAoSEIB2E0EGKFAHpJcCKhaooilLUVX/uKquLuyq7KOqKAgosVcEAopCVJkGKNOkGEkFqqOmFlElm5vfHmECYyeTeyZ3MmeS+z3Mf5dx7P/M9ZU7OnHvO52rMZrMZFRUVFRWXonV1ACoqKioqamesoqKiIgRqZ6yioqIiAGpnrKKioiIAamesoqKiIgBqZ6yioqIiAEJ1xjt27KB58+b8/vvvFdK5//77iY6OZsSIEYwaNYqjR49WSC85OZmXX36Zhx9+mNGjRzNlyhQuXLjgcFxDhw5lxIgRLF26FJPJ5HBcxXrFxxdffKGYVlJSksNaKSkpvPLKK/Tr14/Ro0fz2GOPsX37dtk67du3L/XvDRs28M477zgcV1m6rta7+/6ffvqJgQMHcvXqVZfFU0zz5s159dVXS/5dVFRE165dee655xzSmjNnTsm/lyxZwqeffqpInFUFD1cHcDebN2+mY8eOxMbGMnPmTId1fHx8+O677wDYs2cP8+bNY9WqVQ5pmc1mpk+fzsiRI/noo48ASEhIIDU1lYYNGzocV2pqKq+88go5OTkO5/VuvYqilJbZbGbatGmMHDmSDz/8EICrV6+yc+fOCmtXdfbv388///lPlixZQkREhKvDwc/Pj7Nnz5Kfn4+Pjw/79u2jTp06Dml5eXmxbds2nn32WUJCQhSOtGogzMj49u3bHDlyhHfffZfY2FjFdHNyctDr9Q7ff+DAATw8PHjiiSdK0lq0aEGnTp0qFFfNmjX5xz/+werVq6lK+24OHDiAp6dnqfKKiIhgwoQJLoxKfA4fPsysWbNYuHAh9evXd3U4JfTu3Ztdu3YBEBsby9ChQx3S8fDw4LHHHmP58uUKRle1EGZk/OOPP9KzZ08aNmxIcHAwv/76K1FRUQ5p5efnEx0dTUFBAcnJyRVqAGfPnqVVq1YO32+PyMhIjEYjqamphIaGyr6/OJ/FPPfccwwZMsShWO7WqlevHp999plDOmfPnqVly5YO3WsvJoDMzEz69u2riLZIGAwGpk2bxooVK2jcuLGrwynFkCFD+Pzzz+nTpw+JiYn86U9/4siRIw5pjRs3jhEjRvDMM88oHGXVQJjOODY2lokTJwKWBhAbG+twZ3z3T+5jx47x+uuvs3nzZjQajWLxioCI0xT3Mnv2bI4cOYKnpyfr16+vUEwbNmzg119/VTpEl+Ph4UH79u1Zt24ds2bNcnU4pWjRogVJSUls3ryZ3r17V0grICCA6OhoVqxYgY+Pj0IRVh2EmKbIyMjgwIEDzJo1i759+7JkyRJ++OEHRX6+t2/fnvT0dNLS0hy6v2nTpsTHx1c4DltcuXIFnU5HzZo1naLvCpo2bcrp06dL/v3222+zbNky0tPTXRiV2Gi1Wj7++GNOnTrFwoULXR2OFX379uXf//63w1MUdzNp0iTWr19PXl6eApFVLYTojLdu3Up0dDRxcXHs3LmTn376iXr16vHLL79UWPv333/HaDRSo0YNh+7v2rUrBoOBtWvXlqQlJCRUOLa0tDTefvttxo0bV6VG7F27dqWgoICvvvqqJC0/P9+FEbkHvr6+LFq0iE2bNhETE+PqcErxyCOPMG3aNJo3b15hrRo1ajBo0CDWrVunQGRVCyGmKTZv3syUKVNKpQ0YMIDNmzfTuXNn2Xp3zzWazWb+9a9/odPpHIpNo9Ewf/583nvvPb788ku8vb2JiIjgjTfecDiuoqIidDod0dHRPPXUUw7FdbdeMT179iy1FMkVaDQaPvvsM95//30WL15MSEgIvr6+Lo/LWRQVFeHl5aWIVo0aNVi8eDHjxo0jJCSEfv36OaSTl5dHr169Sv791FNPVaidhYWFlUwhKsHkyZNZvXq1YnpVBY1qoami4jgJCQnMmjVLHempVBghRsYqKu7I119/zcqVKx36laSici/qyFhFRUVFAIR4gKeioqJS3VE7YxUVFRUBUDtjFRUVFQFQO2MVFRUVAVA7YxUVFRUBUDtjFRUVFQFQO2MVFRUVAVA7YxUVFRUBUHfgqbicRYsWER8fT3x8PElJSURERJT7ZpCNGzeyZs0afvvtN8xmMxEREQwePJhp06ZVUtQqohPrKd3YaGhhohMjkUaldsY9hv9kM/3FKY0ZM6IeP/2czIEjadwX6c8jw8I5EZ/JS387yb17BPdu6l2mnghaAycdL5U+eUxdhvcL5cCxTM78novRaKbt/QH07hLM+ct5vPjObxgKS4ttXd5OsbiKY3O2lqNlNm/ePHwDgoi4ryW+6dncLjCz4VDZ7wZc9+WbHN2zkajOA3j4keFotFrSk5O4du2a4m2sz6MHrbTivulCZnYhZy/k0qyRP7m5Rp6YftzquruvB2XbmBJxFd8jqlZ1w+Uj44b1/fjTsAh2/ZzMrPfv+OBev5nHy8815eFetdn+0y231tpzOIM1m2+Sm3eng4mNS+XqzQLGjghjUO+afL8jxa3zWBG91z7cRkjtSAA+/stwCgpyy/yMw7vWcWT3BsY8N4cOPaJLnRv9gNZmh6d0PsdOP871WwUALP2gNb4+jjkCihqXyFpy0Hi6lzWty+eMH+5VG61WwzfflX4b7qat18nLNzLgodpur3X2Yl6pjriYnw5mANAgQvpbD0TNY0X0ijvi8jCbzeza9CXhDVqWdMQFebfLfQmB0vks7lgqiqhxiawlB52vTvIhAi4fGbdoGojRaObMb1ml0g2FZs6ez6FF00C31yqL0BBPANKzilwSl9J5dHaZJV+/QNqty3TrP44fN37Ovq0ryMvJxNs3gLbdhjKw1esuictRRI2rqqD1qOIj419++YXZs2crFkBoiBeZWYUUFlmPblLSCggO8sJDYqGKqmULrQbGjQijqMhM3H7pryQSOY/OLrOU6xcAOHngB/b+sIyHhj3LuJmfENWpP4d2rmXq1Kk2R8nOjstRRI2rqqDx1Eg+REDSyPj06dNs2rSJrVu3EhERwYABAxQLwMdbR2GR7Yc1BoPpj2u05BQZ3VbLFlPHRdCyqT9LY66RdEP6zziR8+jsMivIvw3A7ew0nn59CU2iugMQ1XkAZrOZQ3s3oo/ogrd/60qNy1FEjauq4G4j4zI74wsXLhAbG8vmzZsJDg5myJAhmM1mVq5cqWgA+QVGgn1sv7bGy0v7xzVlP1l3B617mTg6jOj+tYiNS2HtZukPaJSOS+k8OrPMADy9LHPr+uA6JR1xMR16juTo3o0U5v5m1Rk7Oy5HETWuqoIoI16plDlNMXjwYA4cOMCiRYv4+uuvmTBhAlqt8s/7UtIMBOk98bTxVyw0xJv0TANFNn7GuZPW3YwfGca46DC27k7lP8uSZN8vch6dVWbF6IPrABBYI9TqXGCNWgCYTdYrMZwdl6OIGldVQeuhkXyIQJm96/z586lVqxYTJ05k1qxZ7N+/v9yn1o6QcDYbnU7D/c30pdK9PDU0bRRA4rkct9cqZvzIMCaMCmPbnjQ+WnpF9v1Kx6V0Hp1RZncTFtkMD09vstJuWp3LSrsBgFZn/dDL2XE5iqhxVRV0XlrJhwiUGcXDDz/MRx99xA8//ECXLl1Yvnx5yevl9+7dq1gAP+5JxmQy82h0RKn04QPr4uujY9su6y+eu2kBjIuuw4RRYezYl8a8JZdtbqao7LiUzqPSevfi5e1LVOcBZGemEP/L9lLnDvy4xnLNPVMUlRGXo4gaV1VBo9VIPkSg3Ad4fn5+DB8+nOHDh5OZmcmWLVv48ssv6dGjhyIBnL90mw2x13hkeATv/rUl+4+k0aCeH48Mj+DYqQxZi95F1RreL5SJo+tyM8XAsfhs+nQLLnU+I7OQo/HSRkGi5rEiekf3fkdGyjUAbmenU1RUyM6NCwCoERpeanPHwDEvcS5+P2s+f43u/cdRo1YEiSd2k3j8J0aOHMm+042dns/+PUOpU8sy11s8zTB+dDgAN5MNbN8jbQOPqHGJrCUHjU6MEa9UZK0zDgoK4rHHHuOxxx5TNIj/LD7HjVv5jBhYl26da5KZVci6zddYsvqC7BGkiFrNGvoBUCfUi9eevc/q/IkzORyNP1fpcSmt5ajeLz+t50LC4VJp29f/B4CGLTqX6oxrhIbzwttr2BrzMUf2fEt+bg4hdSIZ8sT/8f5bT9Ereo/T8zmkby3atSo9tfD045aNK8fjs2R1LqLGJaqWHLQ6MUa8UnH5pg8AkwnWbExizUb5D7TcQevDxZf5cPHlCsdTjIh5rIjes2+ukPUZwbUiePyFuVbp9h4wK5nPl2efqbBGMaLGJaqWHESZfpCKEJ2xioqKitKI8mBOKhqzM5ZIqKioqLiYI30elHxtx7h9ToxEGurIWEVFpUqiTlPYobr4GYvqWyuqnzFAboGZ62lmwkM0FBTCJxttmydFhGp4sr+OrFw4nGjZnda5uRZ/H/D21AjdxkTzRi6OTVStiqI+wJOJqP68omqBuH66jup9srGQjD9W9j0/zAMvO61ycCctRiMs21ZEdp4lLf6SiWnDy75Jrcuq0/bl4G4jY5fPcIvqzyuqFojrp+uoXobEjWbBARARquX0ZXNJRwyQnQenL1uGVyE1PBWLyxbVoS5F1ZKLRquVfIiAy6MQ1Z9XVC0lcTc/44hQy0jnSoq1eU7SH2nNm1h/hlqX1bPt6zy1kg8RcHkUovrziqqlJO7mZxzoa7k328ZbmbL+SKtV09upcVWHuhRVSy7uth1admeclpamqGGQVE9XVUt5lI7L2fn0/GNa2Ja9b3Gatw19tS6rZ9uvUtMUx48fZ8KECUyfPp3Tp08zbNgwhg8fTvfu3dm9e7ciAeQXGPH0sB2GI/68VV1LSZSOy9n5LPxjgYWHjWdcxWkFNvTVuqyebb9KjYzfeecdnnvuOYYOHcqkSZP45z//yb59+1i1ahXz5s1TJABR/XlF1VISd/Mzzs6z3BvoZ31O/0dacqr1AzG1Lqtn269SnbHRaKRHjx4MHjyY0NBQ2rVrB0DjxtbOWI4iqj+vqFpK4m5+xldTLF/ayFDrZlvvj7TEc9lOjas61KWoWnLReugkHyJgtzO+23jFx6f06+Q1GmX+mojqzyuqlpK4m59xeg5cTTXRsr6GAN876QG+0LK+pT2mZRQ6Na7qUJeiaslFq9NIPkTA7qaPhIQEOnTogNlspqCggA4dOgBgNpsxGAyKBCCqP6+oWiCun66jem0aagjyt3wh/LxBp4WeUZaBQOZtMycv3PkZu+WwiUn9dTw1wINDf+zAe6C5FntjA7Uuq07bl4Mo0w9SsdsZnzlTOdZ3ovrziqolqp+uo3rtm2hpUKf0j7S+7Sw/HS/eNHHywp3lE0kpZpZvN9KnrZa+bbWYgSvJZmL2GHluiPWGD2fkszrUpahachBllYRUXL4dGsT15xVVS1Q/XUf1lm83AtJfR5+UYmblj/JeX6/WZdXQkkNljIynTZvGjh07WLFiBV26WHxDfv75Z+bMmcOlS5cICwvjxRdfZMiQIeVqudefDhUVFRWJOHs1xcaNG8nPzy+VlpSUxPPPP8+ECRM4fPgwf/nLX/jrX//KiRMnyo9X9TNWUVGpilx54U+Sr438fL0s7Rs3bvD444/z1Vdf0adPn5KR8aeffsru3buJiYkpufall17C19eX999/366mEBaaGg2MGRFB9KBwwmr7kJFpIG5vMotXX7S5INyeJeHYkeE0behHs0b+hNfx4catAoctCR2NSzStYj1nazmazx+PGQkL0RAeoiE4UENGjrlMC02AiJoa+rbTWrwqzHAlxcyOY0amDvUUuo2J1l6L8ymqVkWRM2eclZVFVlaWVbper0evLz2nbzabeeONN3j++ecJDw8vdS4hIYGoqKhSaVFRUcTGxpYbgxBzxjOfueN3uubbKxa/0+ERNG0UUKbfaVlMGRtZ4g8b4F+x7CkZV3XQclSvX3tdiZ+xj5d9/bv9jHeduONn/NQA+3UtahtT26v8NiYZGctvly9fzvz5863Sp0+fzowZM0qlffXVV5jNZpsvZs7JyaFJkyal0vR6PTk55a+ndnlnrPrDVg2tiui5k58xKNfGlNQStV24i5/xpEmTGDVqlFX6vaPiy5cvs2DBAtauXWtTJyAggOzs0huPsrKyCAgIKDcGlz/AU/1hq4ZWRfTcyc8YlGtjSmqJ2i7cxc9Yr9dTr149q+PezviXX34hIyOD0aNH06VLl5IVFC+88AJvv/02LVq04Ndffy11T3x8PC1atCg3Xrsj40uXLpGSkkLHjh1LpR85coRatWpRv359SYViD9UftmpoOUPvXsrzM27fREvzJoHs/yWtUuMSAVHbhSvL3hlL2wYPHkz37t1LpfXu3Zt//vOfdO/enaysLBYvXsz69esZMWIEP//8M3FxcSxfvrxcbbsj4/fee8/m8DogIID33ntPZjZso/rDVg0tZ+jdiwh+xqIiartwZdk7w5vC19eXsLCwUgdASEgIQUFBREZGsmDBApYtW0bHjh159913ee+992jbtm252nZHxikpKTRv3twqvXnz5ly9etXGHfKR6neaY8vE1okoGVd10HKG3r0428+4stuYkojaLlxZ9pW1HToxMbHUv7t3786mTZtk69gdGd87EX039y52dhTVH7ZqaDlD715E8DMWFVHbhUvLXquVfgiA3SiioqL45ptvrNJjYmJo1aqVIgGo/rBVQ8sZevcigp+xqIjaLlzqZ6zRSD5EwG5n/MYbb7BhwwYmTJjAnDlzmDNnDuPHj2fdunW8+eabigSg+sNWDS1n6N2LCH7GoiJqu3Bl2Vep1y6FhoayZs0apk2bRkREBBEREUybNo21a9dSq1YtRQJQ/WGrhpYz9O5FBD9jURG1Xbiy7DUeOsmHCEja9NG1a1e6du3qlABUf9iqoVURPXfyMwbl2piSWqK2C9XPWDou34EHqj9sVdFyVM/d/IyVbGNqe5WnJQeNRozpB6kI0Rmr/rBVQ8tRPXfzM1ayjant1Ym42chYtdBUUVGpkmT8a7rka2u8bm0SVNkIMTJWqb5cuHCB77//nn379nH58mUKCgqoX78+gwYNYtKkSfj5lV7Hdv78eT744AMOHz5MYWEhLVu2ZMaMGXTr1s1FOVARFXebM67UkXFZXrMvTrljsXfgSJrFYm9YOCfiM21a7Nnz+hVBy5Zvbdw3XUqsEps18ic31+iwb63cuIpjU1JLiTwCDH1gDytXrqb9Az1p3DwKnc6DM6eOcGjfDiIbNOGtfy/Fy9vyZvKb15OY/dqT6LQeDBjxOH5+AezatpGrl3/nlbc+4enH+1ebNiZqu1BSq6JkzXtJ8rX6P39c4c+rKC4fGYtq1yeq7WJVy+PAgQPp0OcJ/PzveKD0Hfwn6qyKZFPMf/lpx/f0H/ooADErPyP3dg6zP1zBfY2aAfBgnyG8MeNxViz6N5Mfe9jp+VS13MdCU5SddVJxebSi2vWJartY1fLYunXrUh1xMV169Afg6qXfASjIz+P4oT3cH9WhpCMG8PH1o3f/aG5cu8ypU6dsfoaoZaZqOdlC08124Ll8ZCyqXZ+otovVIY8AaamW0ZK+Rk0Arlw8S2GhgcbNW1td27i55TU3ls64ntV5UctM1XJyG1NHxvIQ1a5PVNvF6pBHk9HI92uXoNPp6NZrIADpaZb1tsE1rXd+BodY0m7etL2bS9QyU7Wc28ac/XZopZHcGaelpZGWllb+hTKRarHnzlpKUh3yuHrJPM4lnmLU2OeoW+8+SzwFFpdAT0/rl+R5elk8jPPy8qzOgbhlpmo5uY1ptNIPAbA7TWE2m5k/fz6rVq3CbDZjNpvR6XSMHz+e6dOlr+GzR36BkeAy3kLpiF2fiFpKUtXzuH71QnbExvDQwFEMf+TJO/H8saKisNBgdU+hwTJX7evra3UOxC0zVcu5bUyjE8NzQip2/yQsW7aMo0ePsm7dOg4dOsThw4eJiYnh2LFjLFu2TJEARLXrE9V2sSrn8duvv+D7b5bSs99wnnz+L6XOBYeEApCemmx1X3qaJa1OnTo2dUUtM1XLyW1Mq5F+CIDdzvi7777jww8/JDIysiQtMjKSuXPnsnHjRkUCENWuT1Tbxaqax2+//oKNaxbTo+9QJk9/0+oJd737muDp6cXvidYrJn5PtLwAMioqyqa2qGWmaqkWmndjN4qioiJCQkKs0kNCQigqKlIkAFHt+kS1XayKedy4ZjEb1yym+0ODeXrG39Da+HL4+PrRrnMPzvx6lMsXfitJz8/L5aft31EnPJI2bdrY1Be1zFQtJ7cxjUb6IQB254w9Pct2wbJ3Tg6i2vWJartY1fK4evVqvv36C2rWCqNV2wfYv3trqfNBNUKIamfZjThm4jROn/yFuW/PZOCIJ/D182fXto2kpybz57/NK3O9qKhlpmo510LT3Za22e2MExIS6NChg1W62WzGYLB+kOIootr1iWq7WJXyWLxRIzX5Bl9+MtvqfIuoDiWdcZ26kcya8yXfrPiM2A3LKSos5L7GLXj17U9o1e6BSsunquU6LTm42wM8Ibwp5GJvf74IWrZ8G+Riz5vCEcryDXBUS4k8giWfBxIyFdHq2iKo2rQxUduF0uVfEfJWvSf5Wt/xb1T48yqKy3fgqaioqDgFQVZJSEX1M1ZRUamS5H/9L8nX+jzxuhMjkUaljozL+mk7dmQ4TRv60ayRP+F1fLhxq8Bhe0mNBsaMiCB6UDhhtX3IyDQQtzeZxasv2lxcbu9nnwha0c8nlkqPqOPJY0NCaRTpTUgNDzx0GpLTCjny622+3Z5GepbtN2B8t6C5zfKXW/ZgKf+yfo46ks+JLy0k+eppkpPiyU5LIjA4nPFv7LS6tqiwgN+Ofsel07tIvZ5AbnYqfvpa1Knfhk4PT2P2lCaKxgXK1qUS5a9k2y+ObeCk0p8XUcebft2D6RAVSN3a3nh5arh+y8Cewxls2JpMgcG21tbl7RSNq8K42chYiGmKKWMjS3xwA/wrFtLMZ+54p6759orFO3V4BE0bBZTpnepOWjVreBIcpOPAiRxS04swmszcF+7NgB416NlJz0vvXSQzW/oriZQse3Asnwd/+AhvvyBqRbTCkJddpnZ2+lV+WvcWYQ070qLzI/gH1SYr9Qrx+9dw/tR2BrderGhclaElYtsf2CuE4f1COXAsk5370zEazbS9P4AnH6lLrwdq8OI7v2EolJ5JJctLFoJsc5aKEJ1xVff6VVLrZGIuJxNzrdLjz+Xy+pQI+nYN4tvt0j1ElCp7cDyf4/6yHX1Ny6qLNR8Mp8hw26a+r38wY176ltCI+0ulN+0wnJiPRvHvf/8beFGxuJTMY1mI2Pb3HM5gzeab5ObdGbXGxqVy9WYBY0eEMah3Tb7fIW31j0v9jN1sNYUQfzqqutdvZXi6JqdaNuEE+MmrUqXKHhzPZ3FHXB4+/sFWHTFASJ0mhIQ15ezZs4rG5WwtELPtn72YV6ojLuangxkANIjwcUlcsqlKRkHuhqg+rM7wdPX00ODjrcXLU0NkXS8mjbLYSB6Jtz2qrAxc5V1rNpnIzU4mNDSUQifHJaoHdGXEFRpi2eiVniV9961r/Yzda85YjD8JCiGqD6szPF37PxjEqg+asPT9xsyeGYm/r5Z5S69x+pxtG8nKwFXetfEH1pCblczIkSOdHpeoHtDOjkurgXEjwigqMhO3P12YuOziZiNju1F8+eWXJf//ww8/lDo3b94850RUAUT1YXWGp+vBEzm89ckV3lt4lTWxKdzOMxEY4No5Mld41964eJSfN82hZt0WTJ061elxieoB7ey4po6LoGVTf1Z8e52kG9KnVlzrZ+xe3hR2S+F///tfyf9/8cUXpc7t2bPHORFVgPwCI54etrPkiA+riFrFpGYUcSIhl4Mncvh6cyofL7/Bk6Nq8aeB1sZOlYUz8mmP5KRfiV06FT99bYY8vQhvb2+nx1XZeZSKM+OaODqM6P61iI1LYe1meQ/bXFpeWq30QwDsRnH3fpB794aIuFdEVB/WyvB0vXS1gPNJBQzpVaNCOhWhMr1rk5Pi2fTF03j7BBA9dTkBQba9jJWOSzQPaGfHNX5kGOOiw9i6O5X/LEsSJi5JaHXSDwGw2xnf7YJ1ryOWKG9UvRtRfVgry9PVy1NDgL/rGlZl5dPSEU/G09ufEVNXEBgcYfd6d6xLuTgjrvEjw5gwKoxte9L4aOkVYeKSTFUaGRe7trVv357ExEQ6dOhQ8u/ffvvN3q0uQVQfViW1auhtd7atm/lSP9ybxAuue4BXGd61yVdPs+nLyXh6+xE9dTn6EOu3QTszLlE8oJ0d17joOkwYFcaOfWnMW3LZ4Y0Zqp+xdOwubTtz5kylBFHVvX6V1Hr+iToEB3lwMjGX5NRCPD01NKnvQ49OevLyTfx3vfVrieyhVNlXJJ+JR74jJ/0aAPm30zAaCzmyYwEAAcHhNO8YDVh24G3+YjIFeVm0fnACNy4e48bFYyU635nBbPJFoy09dyxqXYKYbX94v1Amjq7LzRQDx+Kz6dMtuNT5jMxCjsZLG9G61M9YkFUSUhFinXF18PpVSmv34Wz6dNXT5wE9+kAdZjMkpxWydU8G325PIyVd3htYlCx7cCyfCYfWce384VJph7Z+AkB4o84lnXFWWhL5uZaNB79sn2+l8+MaCGn4Hjqt9YM8EesSxGz7zRr6AVAn1IvXnr3P6vyJMzkcjT9X6XHJxkkj3s8//5wNGzaQnp6Oh4cHUVFRvPrqq9x/v2VD0unTp3nnnXc4c+YMwcHBTJ48mYkTJ5arK0Rn/PJs5UbgJhOs2ZjEmo3yHza4g9a+o9nsO1q2f4NclCx7cCyf0c+vlHRdROMuPD83oczzL43QlGkUJGJdgpht/8PFl/lw8WWFolK2vORgdtJ26MGDBzNu3DiCgoIwGAysWrv9G08AACAASURBVLWKKVOmsHv3bnJzc3nmmWcYO3Ysy5cv58yZMzz77LPUrl2bQYMG2dVVLTRVVFSqJHlxqyVf69tnnEOfYTAY+Oqrr3j//fc5dOgQP/74I/PmzWP37t0l73KcO3cup06dYsWKFXa1hBgZq1RvFi1aRHx8PPHx8SQlJREREcHOndYWmmBZUrlmzRrWrl3L+fPn8fLyom3btsyYMYN27dpVcuQqQiNjzjgrK4usrCyrdL1ej16vt0rftWsXr776KtnZ2Wg0Gp566imCgoJISEigZcuWpV6qGxUVRUxMTLkxVGpnfK9vKsDkMXVL7PrO/J5bYtfXu0sw5y/n2bTr27q8nU29imjZ+nn74pQ71n8HjqRZrP+GhXMiPtOm9Z89D1xXahXrKVn+ZU0HOJLPefPm4R8QRP1GLfDLyCLfYGJ3GR4bqxa9x09b19E8qhOjxr+IoSCfPds3MG7ceJYuXcJfPrR5G3HfdCmxqmzWyJ/cXKPDvsGO1qUtP2NXxlUcmxLfIyi7XTgaV0Uxy5gzXr58OfPnWz+HmD59OjNmzLBKf+ihh/jll1/IyMhg48aN1K1bF4CcnBwCA0v7bej1enJyyn/g6fKRsZJ2faJa/4mqBWKU2Xuff0+tMMsStbdfHENBvrVFKMDlC4n8tHUdUe27M3PWpyVr3XsP/BN/mzGat956C7P+72hsjIhEtKoUNS4R2oQiyBgZT5o0iVGjRlml2xoV302NGjWYOHEinTt3plGjRgQEBJCamlrqmqysLAICAsqNweVrP5S06xPV+k9ULRCjzIo74vJI/NWy4qJbn+GlNh35+QfSrnNvLl68iOG2bRtNEa0qRY1LhDahCDLWGev1eurVq2d1lNcZA5hMJoqKirh06RItWrTg9OnTmEx3yi8+Pp4WLVqUq+PyzrgsHLHrU1JLVNvFyrIkdHWZ2aKo0GKQ6eVl3Rl4eVvSCnOlL7lyBNVCU6w2YQ+zTif5kMOKFStITras509LS2P27Nl4eXnRrl07BgwYgNFoZMGCBRgMBk6ePElMTAxPPPFEubpCdsaO2vUpqSWq7WJlWBKKUGa2CI9sBEDCqUOl0s1mM7/FHwXAWCj9LSeOoFpoitUm7OIkC80DBw4wcuRI2rVrx4gRI0hJSWHZsmWEhoYSEBDA4sWL2b17N506dWLGjBlMmzaNwYMHl6trd8742rVrhIeHywpUCYrt+pbGXJNl16ekllTrv5yi8t83J6pWWYhQZraI6vAgdSMbsWtrDDVCatGha18MBfls27Saq1csI2KzyeCQtlQqo/wdwdlxidom7GF20g68zz//3O75li1bsnbtWtm6dqOdNm1ayf/beqLoDCpi16eklqi2i862JBSlzGyh03nw4qxPadKiHetX/oc3p41k9p8f58r5BEaPnwmARit9PtMRVAtNsdqEXaqSN8Xd+0GuXHHMtUkOFbXrU1IrJc1Ag0h/PD00Vj+xHLFdFFHrXkQqs7KoWasur77zBanJ10m9dQ3/wBpE1G9M3A/fAODh49xfcpWRR5Hicoc2URbOGhk7C4ctNJVGCbs+JbVEtV10liWhaGVWHjVr1aVZq45E1G8MwKmje9FqtXgHtlbsM2yhWmi6Pi7JVCU/Y3sWmh06dFAsCKXs+pTUEtV20RmWhCKWmRyOH/qJU0f2Eh0djYdXqFM+oxjVQtO1ccnBrNFIPkTA5RaaStr1iWr9J6oWiFFm+3dtJjX5BgDZWekYiwrZHLMYgJq1wuj20LCSa5d9NhvMZiIbNsfTy5tzZ45zcPcPNGjSijfffJMRT5+2+RkiWlWKGpcIbUIR3GyawuU78JS06xPZ+k9ULRHKbO+P3/Fb/JFSad99bXli3axVx1KdccMmrdi9fQNHD+ykqKiQ2mGRjHh8Kv2Hj7Pahno3IlpVihqXCG1CCcyIMeKVSqW6ttnyRnCEsrwpKqJVlteCHOz5SbhSq1hPyfJXMq6yfCjk0quVv03/B0ew5wEhF3veFHJRMi4Qu11UlIxjts2mbFGjfd8Kf15FcfnIWEVFRcUpuNk0hepnrKKiUiVJO7lH8rUhbXo6MRJpVOrIuKyfMBoNjBkRQfSgcMJq+5CRaSBubzKLV1+0uSDc3k94Vcv2Avq9m3rb/Jk8dmQ4TRv60ayRP+F1fLhxq8CuhSNYfiorWZcvv7WApIunSbpwmrRbSQSHhjPr0+1W16YlX+XdmQPsxhYY9jQ++i6KxAXi1aW9KQ8l69LRNqakVoURZJWEVISYppj5zB2/0zXfXrH4nQ6PoGmjgDL9TlUtZbSmjI0s8dMN8K94c3Aktv+t/Ri/gCAiGrQk77a1wXcx/oHBjH1hjs1zG5a9i7EwHy//lorFpWQeK0NLyboUNY9ycLdNHy7vjEX1+q0OWqCcn25FYnvj4y3UrGNZQTD3tegy/Yy9ffzo2HO4VfrF346Tn5vNwIEDOXrBekWFqOUval2KnEc5uNtqCpf/6RDV67c6aIFyfroVia24I3aUg3HrARgzZoyicbmTFojpjexKP2OzRiv5EAGXRyGq12910FIaV8RWkH+bEwe2EBwazoMPPuj0uETVUpIqk0c3Mwqy2xnv2LGD1avvvGF1zJgx9OvXj379+rFlyxZFAhDV67c6aCmNK2I7vn8LBfm5dO49qtRLIJ0Vl6haSlJV8mjS6CQfImC3M168eDF9+95ZDG0wGFi3bh0rV67k66+/ViQAqX6nqpbyWkrjitgOxq1Ho9HywEMjKyUuUbWUpKrksUpNUxQWFpa89RSgY8eOBAcHEx4eTl5eniIBiOr1Wx20lKayY7uRdI5LZ0/QrHU3gkPLts4UtfxFrcuqkkczGsmHCNjtjLOySs/zvPXWWyX/n5amzOttUtIMJQYp9+KI16+q5Ro/Xaj82A7FbQCgS98/VVpcomopSVXJY5UaGbdp04ZvvvnGKn3NmjW0adNGkQBE9fqtDlpKU5mxFRUZOLJ3EwH6EKI62vcVELX8Ra3LqpJHd7PQtNsZv/HGG2zYsIEJEyYwZ84c5syZw4QJE/j222954403FAlAVK/f6qClNJUZW/yROHKy0ujYYzg6D89Ki0tULSWpKnl0twd4djd91KxZkzVr1rB//37OnbNY5vXu3Ztu3bopFoCoXr/VQQuU89OtSGy/7Pme9ORrAORkpWM0FrJ9w0IAgmuF06nnCKt7pE5RVCQud9ICMb2RXelnLMpcsFQk7cDr1q2boh3wvYjq9VsdtJT003U0tkNxG/j9zOFSaVtiPgWg8f2drTrj9NTrJJ78mQbN2lEnorHT4nI3LRG9kZXWkoMoc8FScfl2aACTCdZsTGLNxoq9hFTVks/Ls5V9m4sjsb3w1jJZnxFcsy4ffHXK6XG5m5aSdSlqHuXgbiNj1UJTRUWlSnLhd+lvI2nYuIkTI5GGECNjFRUVFaVxt5FxpXbGtl7vMnlMXYb3C+XAsUzO/J6L0Wim7f0B9O4SzPnLebz4zm8YCksP3u29KunFKXfs+g4cSbPY9Q0L50R8pk27Pntes45qRT+fWCq9TXM/xgwOIfFCPqnpRRhNZu4L96ZftyDy8k289N5FMrONpe75bkFzxeIqjk1JLSXrMtazeam0+//1f9Sf8jgegf5c/HwV8S/+o+RcxFjrh3kATf82Hf8m9zF1VgIXruQrEhcoW5eitgsl4iqOTcm4KorJ9dY7snD5yHjP4QzWbL5Jbt6dXTixcalcvVnA2BFhDOpdk+93SHvwIKr138nEXE4mWttCxp/L5fUpEfTtGsS326VtohE1j6BMXerbt6TBzEkk/HUuLef+1er81a++t0rziahD26UWn+N7O2Kl4ipGyboUtV2IGpdczG7WGbs82rMX80p9SYr56WAGAA0ifCRruZv1X3JqEQABftKrQeQ8VrgutVraLPwHyVv3cONb6zd9lEW9SX9Coyt7raiSbawsHKlLJbWqQ3uVi7tth3b5yLgsQkMsC/nTs4ok3yO69Z+nhwYfby1enhoi63oxaVQtAI7IeDuy6Hm0hdS6bPTik/g3b8SRR2fK0o+cNJqinNt4BPg7JS5bKFGXSmpVh/YqF1E6WakI2RlrNTBuRBhFRWbi9qdLvq88u742LYPw8NBI2guvpFYx/R8M4rnH65T8+2aKgXlLr3H6nHTTJdHzeC9S69K3QT2avj2Ds//8nLxLV/G9L6LMa++mZp+u+DWK5Mry9UROKn8DiNy4ykKJulRSqzq0V7monbECTB0XQcum/iyNuUbSDelvL5Bq15dTZP3wwZlaxRw8kcPVmwZ8vLU0ivTmgTYBBAbI24opeh7vRWpdtv7s7+ReuMKFj/8rSz9ysuXtHlf+K68zdrSNFaNEXSqpVR3aq1yqVGf8j3/8A40dE41Zs2YpHtDE0WFE969FbFwKazfLm9jPLzAS7ONl85wj1n9KaRWTmlFEaoblJ/HBEzn8fCyHD1+vj7eXlvVbpT0QET2PdyO1LiPGjiD04QfZ33c85iLpUwaewUGEjexPzpnfSd93RPG47KFEXSqpVR3aq1xMZpc/EpOF3WijoqJo1aoVrVq1YufOnSX/X3wozfiRYYyLDmPr7lT+s0z+bh13s/67dLWA80kFDOlVQ/I97pJHOXV5/9y/cOuHnyi4kYxf4/r4Na6Pb32Lp4JHUCB+jevjEWQ9txgxdjg6H2+u/HedU+KSgyN1qaRWdWivcnG3B3h2O+NRo0aVHHq9vtS/R40apWgg40eGMWFUGNv2pPHR0isOabij9Z+Xp4YAf+k//dwhj3Lr0rt2TeoM7UOfhO0lR7edqwCoNy6aPgnbqT/Z+mWj9Z56BJPBQNKqjU6JSy5y61JJrerQXuVSpTrju7E3XVFRxkXXYcKoMHbsS2PekssOm4eIav1XQ2+78bZu5kv9cG8SL0h/ICJqHotxpC6PPDbT6jg1/e8A3NqymyOPzeTm5p2l7gnqGEVQ2/u5uTkOQ3L5P5mVamNK1qWo7ULUuORiNmskHyLg8gd4w/uFMnF0XW6mGDgWn02fbsGlzmdkFnI0XtpfT1Gt/55/og7BQR6cTMwlObUQT08NTer70KOTnrx8E/9dn+ySuJS2N3S0Lm9s2GqVVryaIvf8ZZvnI596BIArS8ufolCyjSlZl6K2C1HjkotJkBGvVOx2xu3bty8ZEefn59OhQwcAzGYzGo2Go0ePVjiAZg39AKgT6sVrz95ndf7EmRyOxks3/BDR+m/34Wz6dNXT5wE9+kAdZjMkpxWydU8G325PIyVd3jpXEfMIytdlWWh9vAl/bCh5l6+RvG1PpcalZF2K2i5EjUsu7vYAz25nfOzYMacH8OHiy3y4+LJieiJa/+07ms2+o9kVjqcYEfMIytZl3qWrVp4VxZjyC9hWq7NL4lKyLkVtF6LGJRdnzQXPnTuXXbt2cf36dfz8/OjVqxevvfYawcF3fnGdPn2ad955hzNnzhAcHMzkyZOZOHGiXV33+tOhoqKiIhFnzRnrdDrmzp3LwYMH+e6777hx4wZ//esdH5WcnByeeeYZevTowaFDh/j444+ZP38+W7Zssaur+hmrqKhUSQ4nZki+tnldLVlZWVbper0evV5v4447xMXF8corr5RM227YsIF58+axe/dutFrLeHfu3LmcOnWKFStWlKlTqQ/wbNnrAWg0MGZEBNGDwgmr7UNGpoG4vcksXn3R5oJwe7aXqpbtBfRlWWgqqeVoPl9+awFJF0+TdOE0abeSCA4NZ9an1kZBaclXeXfmAJufW0yN+lPxC+5ulT52ZDhNG/rRrJE/4XV8uHGrgCemW9uAFhP3TRdAvLp0x3bhqFZFkTPiXb58OfPnz7dKnz59OjNmzLB77/79+2nRokXJvxMSEmjZsmVJRwyWPRsxMTF2dVy+mgJg5jN3/E7XfHvF4nc6PIKmjQLK9DtVtcTTclTvf2s/xi8giIgGLcm7bT06KcY/MJixL8yxeW7DsncxFubjHdja5vkpYyPJzC7k7IVcAvwr1uxFLf/qoCUHOfv6Jk2aZHPvRHmj4v/973/ExMSwatWqkrScnBwCA0tvUtLr9eTk2F+x4/LOWFR/XlVLvteso3pvfLyFmnUsL86c+1o0BfnWXroA3j5+dOw53Cr94m/Hyc/NZuDAgZy6YdsFbOz041y/ZfGgWPpBa3x9HNucIWr5VwctuchZTaHXB5bb8d5LbGwsf//731mwYEGpHckBAQGkpqaWujYrK4uAgAC7ei5/gCeqP6+qJd9r1lG94o7YUQ7GrQdgzBjrXXrFFHfEFUXU8q8OWnJx5qaPmJgYZs+ezcKFC+natWupcy1atOD06dOYTHfG5vHx8aWmMmzh8s5YVH9eVUu+16wrvGsL8m9z4sAWgkPDefDBBxXXvxdRy786aMnFWduhV6xYwQcffMCSJUvo2LGj1fkBAwZgNBpZsGABBoOBkydPEhMTwxNPPGFX1+WdcXl+p8FBXnjYMBlRtcTScoaeFI7v30JBfi6de48q9cDEWYha/tVBSy4ms/RDDu+++y45OTlMnDiR9u3blxzXrl0DLNMUixcvZvfu3XTq1IkZM2Ywbdo0Bg8ebFfX5XPGovrzqlryvWZd4V17MG49Go2WBx4aqZimPUQt/+qgJRdnbfpITEws95qWLVuydu1aWbqSt0PfjZLboUX151W15HvNVrZ37Y2kc1w6e4LmbR4kODRcMV17iFr+1UFLLqIYAEnF5duhU9IMNIj0x9NDY/VTxhF/XlXLNVrO0CuPQ3EbAOjSV/obPiqKqOVfHbTkYnSzztjlc8ai+vOqWvK9ZivTu7aoyMCRvZsI0IcQ1bGvYrrlIWr5VwctubibhabLO2NR/XlVLfles5XpXRt/JI6crDQ69hiOzsNTMd3yELX8q4OWXMxm6YcIuPwBnqj+vKqWfK9ZR/V+2fM96cmWJ9E5WekYjYVs37AQgOBa4XTqOcLqHrlTFP17hlKnlmXusvg1QONHW+aZbyYb2L4nxal5VLUq389YlDd4SMXlnTGI68+rasnTclTvUNwGfj9zuFTalphPAWh8f2erzjg99TqJJ3+mQbN21IloLCmuIX1r0a5V6Z/KTz9u2WxyPD5LcmcM4pZ/ddCSg9wla65GiM5YVH9eVaty9F54a5mszwiuWZcPvjol656XZ5+Rdb09RC3/6qAl73Pda2SsWmiqqKhUSTYflf5GkmEdXD8udX0EKioqKk7A3YaZQvgZvzjljsXegSNpFou9YeGciM+0abFnz9NV1bJtSViW16ySWo7mc9oH1ibg4aFaImpZO6tF1NLR/wEfjv9m4MvvS7u7ffZqDfo8etBmXHHfdCmx0GzWyJ/cXKMkP+M3lpQ2GPL3gRdGeBHoB4cTTNxMN1EnWEvnFlpuZZhZtKmQwns2k733tDcAY16+UObnFTNhRAj9uwXi66Nly54slmwo7f4V81FDwL3ahaNaFUWUJWtScfnIWFS7PlWr8iw0bXEtxcS1FOudWU3qWZrsz78aJMcFylloPtRWR3CghjVxhZw8XxyfiUu3tDzex5MerXXEHXdsa2/Del4M7aVn1aY0Jo2sKe9eQduFay00nSLrNFy+zlhUuz5Vq/IsNKXi5QkdW3iRnmXi9AV5byhWykKzUV0thiLzXR2xhVPnTRQWmenQ1LFOXquBqY+Gcjwhj4Mnbfs520PUduFaC033Wmfs8s5YVLs+VUs8C80Ozbzw9dZwIN7gsi+Qhw5sedqYgUIj1NRr8POWrzu0t57w2p4sWZ9a/sU2ELVduNJC02jWSD5EoMxpirJMggC8vLyoX78+L7/8Mt26datQAOVZ7LVpGYSHh0bS/nVVy3VaztC7l26tvTCZzeyXOUWhJDfTzUTV0FI3RMP1tDv5qBuiwc/b8n2pEaAht0B6HmuHePDooGDWbcsgOb2IWsHyZw9FbRfObhP2EGXEK5Uya92eSZDRaOTs2bO8+uqrbN68uUIBiGrXp2qJZaFZO1hLk3oeJFwqJDXTOS5fUvg53kjL+7Q80deDzQeMJQ/whnbRUWQ046HT4CmzL50ypia3UovYvCvT4bhEbRcutdB0s87YoWkKnU5HixYtGD9+fIUDyC8w4ulhOwxH7PpULddoOUPvbrq3tmxl/vmk60bFABdvmlkTV4SXp4YnB3ry+uPeTOjvwfnrZhKvWPJWICPEnh39adPMly/XpWCswN8YUduFM9tEeZjMGsmHCFRoNcXjjz9e4QBEtetTtcSx0NRqoEsrL3JyTZw4Vyj7fqX59aKJ+EsGwoI1eHtCcqaZ2/nw/AhPjCYzqVnS8uihg0nRNTl2Jo+MbCNhoZavY0iQ5b9+vlrCQj3IyjGRm2+/wxK1XbjSQrNajIyVRFS7PlVLHAvN1o090ftrOXSm0ObDM1dgNsP1NDMXb1o64gBfCK+p4cJ1s9U647Lw8tQSFKijYys/Pn0zsuSYPb0uAL06BfDpm5H062r/rcIgbrtwpYWm0ST9EAGXd8ai2vWpWuJYaHb7Y4pi/ylllqcpjQYY3tUDjQZ2nZC+5K7AYOLDZTetji9jLKZFx87k8uGym/wSX/5SN1HbhWstNN3Lz9jlmz5EtetTtSrPQtMeQf4aWjb04OL1IpubQKSilIWmlwe8MMKT+Esm0rPN+HhpaNNIS71aWrb+UsT569J/GxtNcOCEdUdbvJriZkqRzfO2ELVduNRC082mKVzeGYO4dn2qljwtZ+h1jfJCp9Wwr4IP7pSy0DSaLNMTbRvrCPS1rC1OSjbz3y0Gzl517bdf1HahWmhKQ4jOWFS7PlXL9XpbDxaw9WDFpyeUstA0mmDtLnm7/+SSnF4kycfiXkRtF66y0FRHxioqKioC4G6dsepnrKKiUiX5cof0a6c87Lw4pFKpI+OBk6wtCyPqeNOvezAdogKpW9sbL08N128Z2HM4gw1bkykwWD+02bq8HWDbRlCjgTEjIogeFE5YbR8yMg3E7U1m8eqLNheX27OqdFTr3nxWJI9KaBXrKZHH4nyWZaHpSJkVnDqILqw+2sAaaHQ6TJlpGM6dIu/nbZhzSu9K86jXCN8HB6OrWx+trz+m7EwKLyaQt/cHgme+V6aF5tiR4TRt6EezRv6E1/Hhxq0Chyw0AQJ8oF8HD5pHagnwhZw8OH3JyI6jRvJtTG0XW2iOf/Oa1blV74bb/Pz8AhPPvHOjzOuVrEtR21hFMQmyZE0qLp+mGNgrhOH9QjlwLJOd+9MxGs20vT+AJx+pS68HavDiO79hKJQ+eJ/5zB3v1DXfXrF4pw6PoGmjgDK9U52tpWQeRS4vR/W0AUEYEo9hykoHkwld7Qh8OvTEu1VnMhb9A3NuNgCejVsR+MQMTOnJ5B+Ow5ybg65WOD4deuJ1fwe7cU0ZG1niZxzg73iz9/eB5234GT/QQkeDMK1NP+PySLhQQNzh0qsmihx4+iRie1UyLrm4229+l3fGew5nsGbzTXLz7vwZi41L5erNAsaOCGNQ75p8v0Pak25RfViVzKOo5VURvayV86zSii6dJXDMc3i3607+z1sB8On6MJhNZC79F+a8O5sFjMnXCBg+0W5sIvsZ30o3su9EnkPxFCNqe3Wln7G7dcYu3/Rx9mJeqUov5qeDlrc/NIjwkawlqg+rknkUtbyU1jNlWqwktT5+JWkabx/MRYWY82+Xvjbb+k0h9yK6n7FOB95ejm8+ELW9utLP2GSWfoiAy0fGZREa4glAepb0ZUTu5sPqSB6V1BLKz1jngcbLB42HB7pa4fj1G22599ydt0AX/n4az3qNCYh+irz92yzTFLXD8es/hqLka3jUsj3/qiRS/YxzZfT9D7Ty4cG2ddHpNGTmGDl4Ko+Y7dnkybDhFLW9utLPWN7aBNfvwrPbGV+/fp26devaPBcXF0efPn2cEpRWA+NGhFFUZCZuf7rk+9zJh9XRPCqpJZKfsU+HnvgPfqLk38b0FLI3LKbo8rmStLy9P6D1C8S7/YN4t+lakm44e4qc9V8S8pf/SIqzIijtZ3zuioFDv+ZxM9WIr7eGts19GNAtgBYNvZm9KIUCgxhe0qK0MTkYBfExkYrdzvipp55i8eLF1KtXr1T6unXrWLhwodM646njImjZ1J+lMddIuiF9iOFOPqyO5lFJLZH8jA0JxzCm3AAvbzzCIvFq1hat3z0GOSYTpuwMCs+fwZBwDHPebTzqN8Gncx8C/jRFUowVRWk/478vLD33uvd4HlduFPLoAD0Du/vz/S5pRjqitlfVz1g6dpvNX//6V55++mkWLVpEgwYNAFi0aBGbN29m1apVTglo4ugwovvXIjYuhbWb5U3s5xcYCfbxsnnOER9WpbTupSJ5VFJL6TxWRM+UnVEy91uYeBzDmaMEPfMGeHqRv28LAAHRT+IR2ZiMBX+HIouVpiHxOMa0WwQMrbi3thSK/YyHdfPgyYGWn+1Gk5lfEk0EZJhp1UAny8/YFrF7chjVN5B2zXwkd8aitldnxlUeoswFS8VuZ9y7d2+8vLyYMmUKn332GTExMZw6dYpVq1YRFBSkeDDjR4YxLjqMrbtT+c8y+Vsn3cGHtaJ5VFJLZD9j462rFN24gk+nh8jftwWtPgTvNl3JO7SzpCMuxnD6CFRSZwzK+RmXhdEE6VlGAv2lP18Xtb2qfsbSKbe2u3Xrxvvvv8+ECRO4cuUKy5cvd1pHPGFUGNv2pPHR0isOaYjuw6pEHpXUEt3PWOPhidbXHwCtvoYlTWOjyWorf1GQEn7GZeHpASFBOrJypAuJ2l5d6WdsNpklHyJgtxW3b9+eDh06MGXKFG7fvs2BAwfo1q1bSbpSjIuuw4RRYezYl8a8JZcd/osmsg+rUnlUUktkP2OPBs3R1Y6g6Op5AIwpNzCbjHi1aIfG27fUtd5tu8uKU2kc9TMO8LX9BP+Rh/V46DQcTZA+Nytqe3Wln7G7mcvbnaawTjtvuwAAEvdJREFU91JSpRjeL5SJo+tyM8XAsfhs+nQLLnU+I7OQo/HS/nqK6sOqZB5FLa+K6Okn/4XCiwmYMtLQeHjgUfc+vKI6Yzbkc3tbDADm/FzyD/6Ib7cBBD37NwqO7cGUdxvPyCZ4tX4AY9otdCFlr1kV0c84uk8gTSK9OHO+gJRMIz5eGto286FVY2/OXTawbb/0UaOo7dWVfsYmQUa8UnH5OuNmDS2L+uuEevHas/dZnT9xJoej8ees0stCRB9WJfMocnk5qmfOzcG7dVe0/oFgNmPKTCP/yG7yf96GKSut5Lrc7eswptzAu0NPfHsMBp0npux0Cn75idyfNhHyqvVOvmJE9DM+c8FARG0PenTwI8BXi8ls5maqkW+2ZfHDvhwKZS4/F7G9KhmXXJylHRsby+rVq0lISOD27dskJiaWOn/69Gneeecdzpw5Q3BwMJMnT2biRPs7REGAzvjDxZf5cPFlxfRE9GFVMo8il5ejetlr5ku+tuDYXgqO7ZUdl4h+xkfP5HP0TL4iWiBme4Wq52es1+sZO3Ys+fn5vPnmm6XO5eTk8MwzzzB27FiWL1/OmTNnePbZZ6lduzaDBg2yq6taaKqoqFRJ/vG19D+af3tC/rj04MGDTJw4sdTIeMOGDcybN4/du3ej/ePB8ty5czl16hQrVqywq+fykbGKioqKMzDLeDCXlZVFVlaWVbper0ev19u4wzYJCQm0bNmypCMGiIqKIiYmptx7Xe5nPHlM3RK7vjO/55bY9fXuEsz5y3k27frs+Rm/OOWOXd+BI2kWu75h4ZyIz7Rp12fPz1gELVv+vHHfdCmxg2zWyJ/cXKNdb97ie5TUKsvP2NF85haYuZ5mJjxEQ0EhfLLR9qgmIlTDk/11ZOXC4UTLt61zcy3+PuDtqVE8LtHqsthn+d7vkiPfIyjbg1huHovzqaRWRTEapf/oX758OfPnW0+XTZ8+nRkzZkjWycnJITCwtN+GXq8nJ6f8B54uHxmLatcnqhYoZweptJaj+fxkYyEZf7TV54d54GWnVQ7upMVohGXbisj+w3Uy/pKJacPLvqk61GV1+B7JRc4M7KRJkxg1apRVupxRMUBAQACpqaml0rKysggICCjjjjuoFppupgXK2UEqreVoPjMkruAKDoCIUC2nL5tLOmKA7Dw4fdnyxQup4alYXLYQtS6rw/dILnIsNPV6PfXq1bM65HbGLVq04PTp05jues1IfHw8LVq0KPdel3fGZeFquz5RtUTG2fmMCLVskriSYt3pJP2R1ryJ9WdU57qsSt8juThrB57RaKSgoIDCQsu2/IKCAgoKCjCZTAwYMACj0ciCBQswGAycPHmSmJgYnnjiiXJUBe2MnWXXFxzkhYeHNN9SUbVExtn5DPxjx1p2rvW5rD/SatX0dmpc7lSXVe17JBezWfohh++++442bdrw9NNPA9CmTRvatGnD4cOHCQgIYPHixezevZtOnToxY8YMpk2bxuDBg8vVdXjOeNmyZTz55JOO3m4XEez6RNUSGWfns9ia0tbtxWne3tbji+pal1XteyQXZ+3AGz16NKNHjy7zfMuWLVm7dq1sXYdHxsuWLXP0VrtU1K7P08N2lhyx0BRRS2Scnc/iHWkeNp5xFacV2NCvjnVZFb9HcjEZzZIPEXC4M3bGXhEl7PqKfQfuxRELTRG1RMbZ+czOs9wb6Gd9Tv9HWnKq9QiwutVlVf0eycVkNks+RMDhzlijUXaeRzS7PlG1RMbZ+byaYvnSRIZaN9t6f6Qlnst2alyi12VV/h7JxWw2Sz5EQJKF5r1H+/btuXVLubWBItr1iaolMs7OZ3oOXE010bK+hoC7XDQDfKFlfcvgIC2j0Oq+6lKXVf17JBeTySz5EAHVQtPNtEA5O0iltRzNZ5uGGoL8LZ2pnzfotNAzyjJOyLxt5uSFO1+WLYdNTOqv46kBHhz6YwfeA8212PuhVh3qsjp8j+QiyIBXMi7fgSeyXZ+oWkrZQSqtBY7ls30TLQ3qlP6R1red5YncxZsmTl6486Q9KcXM8u1G+rTV0retFjNwJdlMzB4jzw2x3vBRkbgqQ0up8q8u3yM5GEVxjZeIyztjke36RNVSyg5SaS1wLJ/LtxsB6UubklLMrPxR3lKoql6X1eV7JAdRXqckFZd3xioqKirOwN06Y9XPWEVFpUry/NwMydcueK2GEyORRqWOjG1ZCAKMHRlO04Z+NGvkT3gdH27cKpBkI2jLrk+jgTEjIogeFE5YbR8yMg3E7U1m8eqLNheX27M3rCpxFcemRFzFsSkV19J/1WbFsv9y9Nhxrl+/jqGwkLphYfTs8SCjR0bj41O2wc2m2P8x//MFAHzz1SraduxcpoWmCO3i6X8k2zzv76NhaA8/2jX3JkSvJb/AzNXkIjbuyuXsldKrQ5b8rRagTBsrzue9dpwRdbzp1z2YDlGB1K3tjZenhuu3DOw5nMGGrckUGGxrlWXH6WhcFcXdRsZCTFNMGRtZ4uka4F+xkGY+c8c7dc23VyzeqcMjaNoooEzvVDUu18W1fv16vv3ue7p2eYC+D/VG5+HBiZMnWb5yFbv37uWTDz/A29vabyI1NZWly5bj6+tLXl6eDWWx8lkWNYO0vDahBj5eGvYcz+dmqhFfHw31ansQrJe3DUCpPA7sFVLijbxzf3qJN/KTj9Sl1wM1yvRGdnZccnG3H/1CdMZKebqK6jWrxlU2AwcOZHD/fvj7+5ekDRsymIjwcL5e+w1btm0nevgwq/vmL1hIeN0w7qtfnx/jdtn9DBHyWRbPjNSj08LbX6STmeP4038l8yiqN7Jc3G01hRCubUp5uorqNavGVTatW7cu1REX07tXTwAuXrpkdW7fz/s5cPAQM6ZNK/V6m7IQIZ+2aFbfk2b1Pflhfy6ZOSZ0Wuwa69tDyTyK6o0sF2dZaDoLIUbGSiGq16wal3xSUiwjr+AapR+s3M7N5bOFCxkyaCAtmjdjc2xsuVqi5rN1E8tmj7RMEzMe09O6iRc6rYYbqUVs2pPLgVPSO/3KyKOrvZHlIkonKxUhRsZKIarXrBqXPIxGI6u/XotOp6PPQ6Uf5Cz57zLMJjOTn5wkWU/UfIbVtExvTBoWiL+vliXfZbP0+yyMRpgyUs+Dba3nysvC2XkUwRtZLu5mFGR3ZDx16lS7Ny9cuFDRYCqKqF6zalzyWPjFYs4kJPDUpIlE1qtXkh5/+jT/+2ELr7/6is2pjbIQNZ8+XpZOKL/AzNwVGRRPcR5LNDBnegij+/jz84kCpHQVzs6jCN7IcnG3kbHdzvj48ePUrVuXoUOH0rZtW+GfTuYXGAn28bJ5zpVes2pc0lm+chXfb97MkEEDefzRMSXphYWFfPzpfNq3a2s1Wi4PEfMJYPhjtHgoPp+7nzXl5ps5/puBB9v6EBaq43pK+R2VM/NYUW9kV5W96P3VvdjtjPft28e+ffuIjY1l8+bN9O7dm2HDhtG0adPKik8WKWkGGkT64+mhsfpZ5EqvWTUuaaxc/RVfrVnLgP4PM3P6tFLnNm2OJSnpKs8+/TRXr10rSc/9Y1nbjZs3Cbli2zJStHwWk55l6YRsraIoTvPzkfYT3ll5VMIb2VVlbyxjRC4qdueMdTodvXr14l//+hfffPMN9913HxMmTGDVqlWVFZ8sRPWaVeMqn5Wrv2LVV1/Tv19fXp45w8ov+2ZyMiaTiVlv/53JU54rOfb9vB+AmS+/wogRI2xqi5TPu7lwzbKhI1hvvTSueI1x9m1pHYoz8iiaN7JcqpSfMYDBYGDbtm28+uqrrF69mgkTJtC/f//KiE02onrNqnHZZ9VXX7Pqq6/p17cPf37pRZvL1QY8/DCz/voXq6NN69YA/PmlF5k7d65NfVHyeS/HEg3kFZjo2tob77sM54ICtLRv7s2N1CJupUvrjJXOo4jeyHIxm0ySDxGwO03xf//3f5w9e5ZevXoxffp0mjVr5pQglPJ0FdVrVo2r7LhWr17NytVfUbtWLdq3a0fcrtLbaWsE16Bj+/Y0btSQxo0aWt1/8NAhALo+0Jm2HTvz90+st+OKkE9b5Oab+Wb7bSYNC+TNycHsPZ6PTgd9OvrioYOvtkgfNSqZR1G9keUiimm8VOx2xt9//z2+vr5cvHiRlStXlqSbzWY0Gg1Hjx5VJAglPXVF9JpV4yo7rlOnTgFwKzmZD+Z9ZHW+TesoOrZvLy8QG7g6n2Wx+1g+OXkmBnXzY+RD/pjNZn5PKuKLb7M4lyR9PS8ol0eRvZHlIMr0g1TsdsYJCQmVEoSSnroies2CGldZzJkzh6nPTHb4M1/988u8+ueXy73O1fm0x9EEA0cTDBXWUSqPInsjy/pcN3uAV6V24KmoqKgUYzK7V2es+hmruJxFixYRHx9PfHw8SUlJREREsHPnTrv3bNy4kTVr1vDbb79hNpuJiIhg8ODBTJs2ze59KtWHUdPPSr722/muX65bqSPjsrxmX5xyx2LvwJE0i8XesHBOxGfatNgr9jq1pSeCli0P3LhvupTYLjZr5E9urtFhb2S5cRXH5mwtR/T2burNvHnz8A0IIuK+lvimZ3O7wMyGQ2WPatZ9+SZH92wkqvMAHn5kOBqtlvTkJK5duyZ0G7vXN3jymLolVpVnfs8tsars3SWY85fzbFpVbl3eTrG4imMTVauiVKkdeJWBkhZ7omqBmLaXSufRUb3XPtxGSG3LA7CP/zKcgoLcMj/j8K51HNm9gTHPzaFDj+hS50Y/oLXZGYhaZqJaVYqqJRd3+9HvcqMgJS32RNUCMW0vlc6jo3rFHXF5mM1mdm36kvAGLUs64oK82+V+6UQtM1GtKkXVkovJZJJ8iIDLR8ZKWuyJqqUkIufR2WWWfP0Cabcu063/OH7c+Dn7tq4gLycTb98A2nYbysBWrzs9rupgVSmqllxMxso1fqooLh8ZK2mxJ6qWkoicR2eXWcr1CwCcPPADe39YxkPDnmXczE+I6tSfQzvXMnXqVJujZJHL7F5EsKoUVUsuqrm8TJS02BNVS0lEzqOzy6wg/zYAt7PTePr1JTSJ6g5AVOcBmM1mDu3diD6iC97+rZ0WV3WwqhRVSy6idLJSsdsZz58/v8xzGo1GkWVESlrsiaqlJCLn0dll5ullmUPVB9cp6YiL6dBzJEf3bqQw9zerzljkMrsbUawqRdWSi7utM7Y7TeHn52d1aDQa1q9fz+LFixUJICXNULK//17kWuyJqqUkIufR2WWmD64DQGCNUKtzgTUsr7A3m6xXYohcZsUoYVUpYh5d+T1yt2kKu53x5MmTS45HH32U/Px81q9fz5AhQ9ixY4ciAShpsSeq1v+3d7chTcRxHMC/9DAJ3ZA0Fh6CGURCCTl944t6VUGCaZCagRLsVSj0Zr0s8E2KLxwJHYjiq+Bg2hpCEKPA7AlysUAjIRj4qhdia2dbkJu9GIi3J3fXXf5z3w/szd3tx91x/Dj+d//vmUnkY7T6nB2vPYVDh8sQW89O+oqtfwMAHDiY/UBI5HMGiBdVKWotvf631LZdH+BFo1GMjY2hvb0dyWQSfr8fHo8HVVVVpuyAmRF7otYyk8jHaPU5s5UdwZmWS1B/rGF5MahZ9/6Fkt4mY4jC7P0qhahKUWvplUomi/6JoOCY8cjICILBILq6ujA3N6fru2PFMjNiT9RagJixl2Yfo9F6H18HEF1Lf73jp/odm5u/8fKpDACorK7RTO64fP0Ovi6/g/LIg9aLN1F5TMLKp1dYCc+jo6MDbz6ftPQ4SyGqUtRaeu2rCM3p6WnYbDbIsqz5+KjZEZpmRuyJWkvU2Euz4w2N1Fucn0XkywfNsuDsQwDAidMtmmZcWV2D2/cVPPd5EVrw41d8A0edtbhy4y4e3LuF81cXLD/OUoiqFLWWHlYOP6RSKXi9XszMzCCRSKCpqQlDQ0OQJGn3P+fxT4OC8uUG6FUoN0CEWrmyKfQqlE1hRKE8ib2uVSiHQo9806GNsOK6yMymMKJQNoURIl8Xf+vCtbdFbzv/pHX3jXaYmJiAoiiYnJyE0+nE8PAwwuEwAoFAzi/VFGPPJ30QEVlhaytV9E8vRVHgdrtRX1+P8vJyeDweRCIRhEIhw/vLCE0i2pf03KU/e3wOsVgsa7nD4YDDoR1eVFUVzc3N8Pl8aGxs3F7e1taG7u5u9PX1GdrfPZ+BR0RkBT1DHePj4zknuQ0MDGBwcFCzbGMj/VA1s0nb7fbtdUawGRNRyevv70dnZ2fW8syGCwAVFRUA0nfIO6mqur3OCDZjIip5uYYj8rHb7ZAkCUtLSzh7Nv1eu6qqWF1dRUNDg+F94AM8IiKdenp6MDU1hUgkgng8jtHRUdTV1cHlchmuyTtjIiKd3G43VFVFb28vEokEXC4XZFk2/FobwLcpiIiEwGEKIiIBsBkTEQmAzZiISABsxkREAmAzJiISAJsxEZEA2IyJiATAZkxEJAA2YyIiAfwBqfHOxdPf38YAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r6yjwuF7IViH"
      },
      "source": [
        "## **Model 7** : LSTM (variable-length inputs + word2vec word-embeddings + frozen word-embedding weights )"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AyFp_FfyKCel",
        "outputId": "b52b5412-590d-4e78-c709-7ebfd1608bd8"
      },
      "source": [
        "w2vec_variables_freeze = LSTM_w2vec2_variable_size(vocab_size,\n",
        "                         embedding_dim = w2vecembedding.vector_size,\n",
        "                         hidden_dim = 50,\n",
        "                         vec_weights = pretrained_weights,\n",
        "                         freeze_emd = True)\n",
        "train_model(w2vec_variables_freeze, epochs=200, lr=0.01)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train loss 2.551, val loss 2.507, val accuracy 0.168\n",
            "train loss 2.066, val loss 2.159, val accuracy 0.293\n",
            "train loss 1.573, val loss 1.652, val accuracy 0.456\n",
            "train loss 1.259, val loss 1.538, val accuracy 0.467\n",
            "train loss 0.886, val loss 1.407, val accuracy 0.541\n",
            "train loss 0.764, val loss 1.287, val accuracy 0.619\n",
            "train loss 0.550, val loss 1.224, val accuracy 0.640\n",
            "train loss 0.295, val loss 1.242, val accuracy 0.643\n",
            "train loss 0.172, val loss 1.263, val accuracy 0.685\n",
            "train loss 0.101, val loss 1.402, val accuracy 0.688\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 664
        },
        "id": "WEwxaYZDOzcF",
        "outputId": "6268d7dc-06d1-45e5-ca72-188793ff2e5b"
      },
      "source": [
        "print(\"***Model 7 : LSTM (variable-length inputs + word2vec word-embeddings + frozen word-embedding weights )***\")\n",
        "pred_vs_true_m7 = matrix_pred(w2vec_variables_freeze, val_dl)\n",
        "diagnostics(w2vec_variables_freeze, val_dl)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "***Model 7 : LSTM (variable-length inputs + word2vec word-embeddings + frozen word-embedding weights )***\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           A       0.73      0.50      0.59        22\n",
            "           B       0.67      0.67      0.67        12\n",
            "           C       0.60      0.69      0.64        13\n",
            "           D       0.64      0.73      0.68        22\n",
            "           E       0.81      0.65      0.72        20\n",
            "           F       0.64      0.79      0.71        29\n",
            "           G       0.73      0.84      0.78        19\n",
            "           H       0.67      0.72      0.70        65\n",
            "           I       0.40      0.22      0.29        18\n",
            "           J       0.74      0.81      0.77        21\n",
            "           K       0.75      0.74      0.74        61\n",
            "           L       0.94      0.94      0.94        17\n",
            "           M       0.64      0.57      0.60        37\n",
            "           N       0.81      0.89      0.85        19\n",
            "\n",
            "    accuracy                           0.70       375\n",
            "   macro avg       0.70      0.70      0.69       375\n",
            "weighted avg       0.70      0.70      0.69       375\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEDCAYAAADgNmW7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydeXhTVeL+P0m6N00XCi0NRUBZhLKVHREEZBcKKCK7G27g9qg/x2XGgXGbLwM6yogoICAoWJYi7SiClFX23UIRZC1L6b7QNmnT/P7otBCSpjfpTXPa3s/z3Ad6b/L2PeeenJ6ce+57VWaz2YyCgoKCgltRu9uAgoKCgoLSGSsoKCgIgdIZKygoKAiA0hkrKCgoCIDSGSsoKCgIgNIZKygoKAiAUJ3xli1baN26NX/++We1dO69915iYmIYNWoUY8aM4fDhw9XSS0tL49VXX+XBBx9k7NixTJ8+nfPnzzvta8SIEYwaNYolS5ZQWlrqtK9yvfLtq6++kk0rJSXFaa309HRee+01Bg4cyNixYxk/fjybN292WKdz584WP69bt47Zs2c77asyXXfr3f7+7du3M2TIEK5cueI2P+W0bt2a119/veLnkpISevbsybPPPuuU1scff1zx8+LFi/n8889l8VlX8HC3gduJj4+nS5cuJCQk8NJLLzmt4+Pjw4YNGwDYuXMn8+bNY8WKFU5pmc1mZs6cyejRo/nkk08ASE5OJiMjg+bNmzvtKyMjg9dee438/Hyny3q7XnWRS8tsNjNjxgxGjx7N3LlzAbhy5Qpbt26ttnZdZ8+ePbz//vssXrwYvV7vbjv4+flx5swZioqK8PHxYffu3YSFhTml5eXlxS+//MIzzzxDSEiIzE7rBsKMjG/evMmhQ4f44IMPSEhIkE03Pz8fnU7n9Pv37t2Lh4cHEyZMqNjXpk0bunbtWi1fDRo04B//+AcrV66kLt13s3fvXjw9PS3qS6/XM2XKFDe6Ep8DBw7w7rvv8uWXX9K0aVN326mgX79+bNu2DYCEhARGjBjhlI6Hhwfjx49n2bJlMrqrWwgzMv7111+5//77ad68OcHBwfz+++9ERUU5pVVUVERMTAwGg4G0tLRqNYAzZ87Qrl07p99vj8jISEwmExkZGYSGhjr8/vJylvPss88yfPhwp7zcrtWkSRP+85//OKVz5swZ2rZt69R77XkCyMnJYcCAAbJoi4TRaGTGjBksX76cu+++2912LBg+fDhffPEF/fv35/Tp0zz88MMcOnTIKa1JkyYxatQonn76aZld1g2E6YwTEhKYOnUqUNYAEhISnO6Mb//KfeTIEd58803i4+NRqVSy+RUBEacp7mTWrFkcOnQIT09P1q5dWy1P69at4/fff5fbotvx8PCgc+fOrFmzhnfffdfddixo06YNKSkpxMfH069fv2ppabVaYmJiWL58OT4+PjI5rDsIMU2RnZ3N3r17effddxkwYACLFy/mp59+kuXre+fOncnKyiIzM9Op97ds2ZKkpKRq+7DF5cuX0Wg0NGjQwCX67qBly5acPHmy4uf33nuPpUuXkpWV5UZXYqNWq/n00085ceIEX375pbvtWDFgwAD+7//+z+kpituZNm0aa9eupbCwUAZndQshOuNNmzYRExNDYmIiW7duZfv27TRp0oSDBw9WW/vPP//EZDIRFBTk1Pt79uyJ0Whk9erVFfuSk5Or7S0zM5P33nuPSZMm1akRe8+ePTEYDHz33XcV+4qKitzoqHbg6+vLwoUL2bhxI7Gxse62Y8EjjzzCjBkzaN26dbW1goKCGDp0KGvWrJHBWd1CiGmK+Ph4pk+fbrFv8ODBxMfH061bN4f1bp9rNJvN/POf/0Sj0TjlTaVSMX/+fD788EO+/vprvL290ev1vP322077KikpQaPREBMTwxNPPOGUr9v1yrn//vstliK5A5VKxX/+8x8++ugjFi1aREhICL6+vm735SpKSkrw8vKSRSsoKIhFixYxadIkQkJCGDhwoFM6hYWF9O3bt+LnJ554olrtLDw8vGIKUQ6efPJJVq5cKZteXUGlRGgqKDhPcnIy7777rjLSU6g2QoyMFRRqI99//z3ffvutU9+SFBTuRBkZKygoKAiAEBfwFBQUFOo7SmesoKCgIABKZ6ygoKAgAEpnrKCgoCAASmesoKCgIABKZ6ygoKAgAEpnrKCgoCAASmesoKCgIADKHXgKCgp1kgRP6cFGI4pPu9CJNGq8My40mLmeBeHBYCiBBQm2nwE3opuKdk1VpOWW/RwerGJloolLafDWo2WhP/0f3Wf1vsQfepCTV8yZ8wW0auFPQYGJCTOPVuon8YceAPQZud3q2MvT72bcqCZs/y2NvYcyuSvSn0ceiuBYUg6v/PU4d967uGtjWd7ruFerfj7elFEhDOoVgK+Pmp935rJ4XYbF8dhPmsvmq9ybHPVV/h5bvpzxJrcvW1rO6JW3Cznb2JBplq95clxjRg4MZe+RHE79WYDJZKbjvVr69Qjm3KVCXp79B8ZiywrbtKwTIG+7EFWrvlGjnfGCBBPZN8v+//QQNZ52fvv23838fMiMqRS6t1YRHiwtZnLizKNcu2EAYMm/2uPr41xaW/Omfjz8kJ5tv6Xx7ke38nmvpRby6rMtebBvIzZvv+GcdhMvRvTVsWJjJtNGO5ZlLLcvuepLbm9y+pJbTy6tnQeyWRWfSkHhrQFJQmIGV1INTBwVztB+DfhxS7okLTnrXlQtR1F51q5o2hqdMy7viKWQXwgmJx6cXP4hqS4P9m2EWq3ihw2WT+nduOkahUUmBj/QyCldtQqeezSUo8mF7Dte4HZfctWX3N7k9CW3nlxaZy4UWnTE5Wzflw1AM730p2HIWfeiajmKxlcjeRMBZc64Etq0DMBkMnPqj1yL/cZiM2fO5dOmZYBTuiP66Yho5Mm/vnFuNOAqX3IgsrfaRGiIJwBZuSWS3yNn3Yuq5Shqjzo+Mj548CCzZs1yhRehCA3xIie3mOIS6wmt9EwDwYFeeDh4shuFePDo0GDW/JJNWpb0D5qrfcmFyN5qC2oVTBoVTkmJmcQ90h9VJWfdi6rlKCpPleRNBCSNjE+ePMnGjRvZtGkTer2ewYMHu9qX2/Hx1lBcYnuexGgs/d9r1OSXmCRrTh/XgBsZJcRvyxHKl1yI7K228NwkPW1b+rMk9iop16VPh8hZ96JqOUptGxlX2hmfP3+ehIQE4uPjCQ4OZvjw4ZjNZr799tua9Oc2igwmgn1sP07Hy0v9v9dIn9S+v4s/HVr58t78a07NhbvKl5yI7K02MHVsODGDGpKQmM7qeMemseSse1G1HEWUEa9UKp2mGDZsGHv37mXhwoV8//33TJkyBbW6/twjkp5pJFDniaeNv66hId5k5RgpsfHVyxYeGpgW04AjpwrJzjMRHupBeKgHDUPK/hb6+aoJD/XAz6fq+pXTl9yI7E10Jo8OZ1JMOJt2ZPDZ0hSH3y9n3Yuq5ShqD5XkTQQq/fTPnz+fhg0bMnXqVN5991327NlDfXooSPKZPDQaFfe20lns9/JU0bKFltNn8yVreXmqCQzQ0KWdH5+/E1mxzZrZGIC+XbV8/k4kA3tqa9SX3IjsTWQmjw5nyphwftmZySdLLjulIWfdi6rlKBovteRNBCp18eCDD/LJJ5/w008/0aNHD5YtW1bxePldu3bVpEe38OvONEpLzTwao7fYP3JIY3x9NPyyLVWylsFYytylqVbb17Fla0iPnCpg7tJUDiZVvdRNTl9yI7I3UZkUE8aUMeFs2Z3JvMWXbN4AIQU5615ULUdRqVWSNxGo8gKen58fI0eOZOTIkeTk5PDzzz/z9ddf06dPH4d/WdRdKnR+/9P1BrUaet9bVhG5BfD7xVstsWEgtIwoO9YkVFXx/iah9n/HoPtDCWtYNkdV/vVo8tgIAFLTjGzeKW0R/bmLN1mXcJVHRur54K227DmUSbMmfjwyUs+RE9kOLVQ3lcLeY9YdbcPgsupPTS+xedzVvkC++pLbm5y+5NaTS2vkwFCmjm1MarqRI0l59O8VbHE8O6eYw0nSRo5y1r2oWo6i0ogx4pWKQ+uMAwMDGT9+POPHj3fql3VoruKuRpZ/hfq1L/v54g2zRWccHqyiX3vLyuzYourKHT6gIZ3aWX4leuqxSACOJuU69KH7bNFZrt8oYtSQxvTq1oCc3GLWxF9l8crzTo9g5EBOX3LWl5ze5PYlp55cWq2al41MwkK9eOOZu6yOHzuVz+Gks5J9ydkuRNVyBLVGjBGvVGr0po/vtkm/anrigpkTF2wvdynPprDFq7NOOeyrMkpLYVVcCqviHL+gIoW0rBJJORZ3IqcvOesL5PMmty859eTSmrvoEnMXXZJFC+RtF6JqOYIo0w9SUe7AU1BQqJOIcmFOKipzfVoioaCgUG841P8+ya/tkrjbhU6koYyMFRQU6iTKNIUd5M6afXW+5ZVmFTBzrC/NwtUcSC7hYqoJTw8V0S09uCtcw6+HjMTvMVq855OZZWt75cwzFk2rXM/VWs6WM+Z562DvDq39GDcshNPni8jIKsFUauauCG8G9gqksKiUVz68QE6e5TWFDQtay+oLxDuXtbFduCvPWLmA5wRy5cM2DVfTIkLD9qNG4nbd6nR3nyjmrUl+9I7ytOqMK0PUTFdRteTWO366gOOnrZf7JZ0t4M3pegb0DGT95swa96Vo1aI841o2MhZihluufFgfr7LKz7lp+afWVAr5RWYMxdKnx0XNdBVVyxV6tkjLKEu70/pJb7qi1pmi5do8Y5VaLXkTASFGxnJxKdVEQZGZAdFeZOaWcjG1FC8P6HavJ5EN1cRuk97pi5rpKqqWK/QAPD1U+Hir8fJUEdnYi2ljGgJwKEn6kwpErTNFy7V5xhpPMTpZqdSpzrjQAIsTChk/wIfHh/lW7C8ymvnmpyJ+Py89pq+qHNYObQPx8FBJCjmpD1qu0AMYdF8gzz4WVvFzarqReUuucvJsoWQNUetM0XKuTUiltk1TONwZZ2ZmEhwcjEolZkENxXAts5Tfz5dw4boJP28Vfdp7MmWwD4v/W8Qfl6V1yKJmuoqq5Qo9gH3H8rmSasTHW02LSG+6d9ASoHXsmoKodaZouTbPWJTpB6nYdXv06FGmTJnCzJkzOXnyJA899BAjR46kd+/e7Nixo6Y8SqZxAzUvPeLLH5dK2PibkRPnTOw7VcJn6wrJKzAzvr83Uv+GFBlMeHrYrh5nMl3rupYr9AAysks4llzAvmP5fB+fwafLrvP4mIY8PCTELb4ULfe2MUeobUFBdjvj2bNn8+yzzzJixAimTZvG+++/z+7du1mxYgXz5s2rKY+S6dfREy8PFUf/tHykUXEJnLxYQohOTUiAtIoXNdNVVC1X6Nni4hUD51IMDO8b5BZfilbtyTOuU52xyWSiT58+DBs2jNDQUDp16gTA3XffXSPmHCVQW1apahvD3/J9Ur+5iJrpKqqWK/Qqw8tThdZf+lSFqHWmaLk2z1jtoZG8iYDdrun2J3v4+Fg+NlzEOePUzLKvO93bWE6F+3hBVAsNN4vMpOdI+yssaqarqFpy6wXpbH9A2rfypWmEN6fPS7+AJ2qdKVquzTNWa1SSNxGwewEvOTmZ6OhozGYzBoOB6OhoAMxmM0ajtJsnpCBXPuz2Y8V0bePJiN5eNG6g5vw1E34+Knq28yTQX82abUWSI/tEzXQVVUtuvecnhBEc6MHx0wWkZRTj6aninqY+9Omqo7ColG/WprnFl6JVi/KMBZl+kIrdzvjUKXljDCtDrnzYrDwzn/xQwJDuXrRsoqFzSw+KTXAlrZQNuwo5cc6xK7aiZrqKqiWn3o4DefTvqaN/dx26AA1mM6RlFrNpZzbrN2eSnlVStYgLfCla7m9jUqltqymEWGcsZ9ZsRq6Z77bIc0efqJmuomrJqbf7cB67D+fJ4gnErTNFy3XUxMh4xowZbNmyheXLl9OjR1lmzm+//cbHH3/MxYsXCQ8P5+WXX2b48OFVatWuPx0KCgoKEnH1aoq4uDiKioos9qWkpPD8888zZcoUDhw4wF/+8hfeeustjh07VrVfJc9YQUGhLnL5hYclvzbyi7UOaV+/fp3HHnuM7777jv79+1eMjD///HN27NhBbGxsxWtfeeUVfH19+eijj+xq1ug0RWXxhioVjBulJ2ZoBOGNfMjOMZK4K41FKy/YXBBuL0bQWa23F1tPbWh9YGC0B60j1Wh9Ib8QTl40seWwiSIb1y8/fMobsB0VOnF0BC2b+9GqhT8RYT5cv2GQFBMqRxnLy+lqLWf0RNYCeduYqO1iyDTL36cP82Zg72CiowJo3MgbL08V124Y2Xkgm3Wb0jAYbWttWtZJVl/VxZE549zcXHJzc63263Q6dDrL61lms5m3336b559/noiICItjycnJREVFWeyLiooiISGhSg9CzBm/9PStvNNV6y+X5Z2O1NOyhbbSvFNXa/n7wPOjvAjwgwPJpaRmlRIWrKZ7Gw3NwtUs3FhMsQPXA6dPjKzIbNb6V6/aRawv0b2JqiViuxjSN4SRA0PZeySHrXuyMJnMdLxXy+OPNKZv9yBenv0HRgcSEOVuY5JxYPntsmXLmD9/vtX+mTNn8uKLL1rs++677zCbzTYfzJyfn88999xjsU+n05GfX/V6ard3xqJmpz7QUUNwgIpVicUcP1f+17uUizfUPNbfkz7tNSQeld4by5XZLGp9iexNVC0Qs13sPJDNqvhUCgpvjVoTEjO4kmpg4qhwhvZrwI9bpK1wqi15xtOmTWPMmDFW++8cFV+6dIkFCxawevVqmzparZa8PMsLz7m5uWi12io9uP0CnqjZqS0aqzGWmG/riMs4ca6U4hIz0S0d+9DIldksan2J7E1ULRCzXZy5UGjREZezfV82AM30PlbHasKXoziSZ6zT6WjSpInVdmdnfPDgQbKzsxk7diw9evSoWEHxwgsv8N5779GmTRt+//13i/ckJSXRpk2bKv3aHRlfvHiR9PR0unTpYrH/0KFDNGzYkKZNm0qqFHuImp3qoQFbQVJmoNgEDXQq/LyhQJ7PkmRErS+RvYmqJSc14Ss0xBOArFzpa7zdWV+uWNo2bNgwevfubbGvX79+vP/++/Tu3Zvc3FwWLVrE2rVrGTVqFL/99huJiYksW7asSm27I+MPP/zQ5vBaq9Xy4YcfOlgM21SVdxoc6IWHjZARV2ulZpnx81bROMTy9Y1DVPh5l+0L0tb8HT6i1pfI3kTVkhNX+1KrYNKocEpKzCTuyRLGlz1ckU3h6+tLeHi4xQYQEhJCYGAgkZGRLFiwgKVLl9KlSxc++OADPvzwQzp27Filtt2RcXp6Oq1bt7ba37p1a65cuWLjHY4janbqb0km2t6lZsIAD+L3miou4I3ooaHEZMZDo8LTDTPuotaXyN5E1ZITV/t6bpKeti39WRJ7lZTr0r8OujfPuGb+KJ4+bflA3d69e7Nx40aHdex2J3dORN/OnYudnaXIYCLYx8vmMWeyU+XSupBqZlViCQ/18uDxIWVfz0ylZg6eLkWbbaZdMw0G+eI5JCNqfYnsTVQtOXGlr6ljw4kZ1JCExHRWxzt2sc2t9VXLboe26zYqKooffvjBan9sbCzt2rWTxYDI2am/Xyjln6uMfL7eyFfxRj7+3siG30rQ+aswlZrJyK35+2VEri9RvYmqJSeu8jV5dDiTYsLZtCODz5Y6fjuzW/OMVSrJmwjY7Yzffvtt1q1bx5QpU/j444/5+OOPmTx5MmvWrOGdd96RxYDo2almM1zLNHMh1czNItD6QkQDFeevmR1aZywXIteXqN5E1ZITV/iaPDqcKWPC+WVnJp8suSyML6nUtqdD23URGhrKqlWrmDFjBnq9Hr1ez4wZM1i9ejUNGzaUxUBtyk5VASN7eqBSwbZjjqWGyYXI9SWqN1G15ERuX5NiwpgyJpwtuzOZt/iS0zdmuLO+VB4ayZsISLoE1bNnT3r27OkSA6Jmp3p5wAujPEm6WEpWnhkfLxUdWqhp0lDNpoMlnLvmWOuUK7NZ1PoS2ZuoWiBmuxg5MJSpYxuTmm7kSFIe/XsFWxzPzinmcJK0Ea2SZywdt9+BB2Jmp5pKy6YnOt6tIcC3bG1xSpqZb342cuaK48MEuTKbQcz6Et2bqFoitotWzf0ACAv14o1n7rI6fuxUPoeTzta4L0dRqcSYfpCKEJ2xiNmpplJYvU2+qQg5M5tFrC9X6NUHLRHbxdxFl5i76JJMrtyXZ0wtGxkrEZoKCgp1kux/zpT82qA3rUOCahohRsZ1iYULF5KUlERSUhIpKSno9Xq2bt1q9z1xcXGsWrWKP/74A7PZjF6vZ9iwYcyYMaOGXCso1D2UOWM7VJY1+/L0WxF7ew9llkXsPRTBsaQcmxF79vJhE3/oURFJ2KqFPwUFJqfzYZ3xNW/ePHy1gejvaotvVh43DWbW7a98Ufuar9/h8M44oroN5sFHRqJSq8lKS+Hq1auy+Sr3JqdWzPOnrfZ3aO3HuGEhnD5fREZWCaZSM3dFeDOwVyCFRaW88uEFcvIs1wNuWNBalvNY/p760Mbk0irXE1Wruqg0YqySkIrbR8Z1LZJwy5YtHLhWtozn07+MxGAoqPR3HNi2hkM71jHu2Y+J7hNjcWxsd9sXH0SOgzx+uoDjp63Lm3S2gDen6xnQM5D1mzMlacl1HqHutbH6quUwgqwflorb3da1SMLIyEhJ+mazmW0bvyaiWduKjthQeJOqpvBFjoOsjLSMsguhWj/pzU2u8wh1r43VVy1HqW134Ll9ZFxfIwnTrp0n88Yleg2axK9xX7B703IK83Pw9tXSsdcIhrR7E39/f5f6clUZPT1U+Hir8fJUEdnYi2ljym4QOpR00ym96lIf2lh90HKYWjYydntnXFXEXoe2gXh4qGr8fn9X+0q/dh6A43t/wmQqpv+o5whp1ITkI9vYv3U1zz13nuXLl7vUl6vKOOi+QJ59LKzi59R0I/OWXOXk2UKHdOSiPrSx+qDlKHX2Al5mZtlcX0hIiKwG6mskoaGobJR4My+Tp95czD1RZYHVUd0GYzab2b8rjh07drjUl6vKuO9YPldSjfh4q2kR6U33DloCtO67mFIf2lh90HKYunTTh9lsZv78+axYsQKz2YzZbEaj0TB58mRmzpS+hs8e9TGSEMDTq+zRNbrgsIqOuJzo+0dzeFcc+/fvB7q7zJerypiRXUJGdtk88b5j+fx2JJ+5bzbF20vN2k3SLuDJSX1oY/VBy1Fq22oKu386li5dyuHDh1mzZg379+/nwIEDxMbGcuTIEZYuXSqLgfoWSViOLrjsa3xAUKjVsYCgsjlWW48Or41xkBevGDiXYmB436BqazlDfWhj9UHLYdQq6ZsA2O2MN2zYwNy5cy1WCERGRjJnzhzi4uJkMVCfIglvJzyyFR6e3uRmWqdW5WZeB2xPCdXWOEgvTxVaf/eMVOpDG6sPWo5SpyI0S0pKbHYIISEhlJTIk9tQXyIJ78TL25eoboPJy0kn6eBmi2N7f10FlD3o0JW+5C5jkM52Z9u+lS9NI7w5fd49F/DqQxurD1oOo1JJ3wTA7pyxp6enU8ccoa5FEsbFxbF1f9maypt5WZSUFLM1bgEAQaERFjd3DBn3CmeT9rDqizfoPWgSQQ31nD62g9NHtzN69Giio6MByzuaRI6DfH5CGMGBHhw/XUBaRjGeniruaepDn646CotK+WZtmmQtuc6jK8rp7jZWX7UcRpARr1TsdsbJycn/6xAsMZvNGI3yPQCuLkUSrl279n8X3m6xee1nADRv082iMw4KjeCF91axKfZTDu1cT1FBPiFhkQyf8P/46G9PyOqrJrR2HMijf08d/bvr0AVoMJshLbOYTTuzWb85k/Qs6d+m5DyPULfaWH3WcoTadgHPbmd86pR88X72qEuRhN9++63dLIo7CW6o57EX5ljtV9v5qy5qHOTuw3nsPlz5Q2wdQc7zCHWrjdVnLYeoS0vbFBQUFGotgqySkIqSZ6ygoFAnKfr+n5Jf6zPhTRc6kYYQEZoqFYwbpSdmaAThjXzIzjGSuCuNRSsv2FwQbi9GsC5pLVy4kI1bk0i9nERuRgoBIXqees92NvKuH//FlT8Pkp1+EWNhHr4BDWgY0YboAU8S2bIHzw+Vx1e5NznP5ZBp1vGT+jBvBvYOJjoqgMaNvPHyVHHthpGdB7JZtykNg9Faa9OyTkK3sTvL6WwZ5fJV7u3OOFR9mCfjh4fSItKbkCAPPDQq0jKLOfT7TdZvziQr1/bdchsWtJbVV7WpZSNjIaYpXnr6Vt7pqvWXy/JOR+pp2UJbad5pfdCaN28ePn5BNGrSFkOh/bnYaxePERrRins6DsbHT8fN3HSSD/7I2vlTGTL5nzB0tMvLKKfekL4hjBwYyt4jOWzdk4XJZKbjvVoef6QxfbsH8fLsPzAWSzfn7nNZm8rYIMiT4EANe4/lW+RSD+4TxP1ddTZzqWvCl8Moc8aOIWp2qghaW7ZsIT6p7Cr9tx89hNFYeTbyuBe/tdrXqd8Uls4exIHNX8FfrTtjubNm5dTbeSCbVfGpFBTeGjklJGZwJdXAxFHhDO3XgB+3SFu1IMK5rE1llDOX2q15xrVsNYXb/3SImp0qgpbUbOTK8PL2x8c/iKJC69uqq+OrMuTUO3Oh0KKTKmf7vmwAmul93OKrPpSxMpzJpXZnnjEqtfRNANw+MhY1O1VUraoozM/EbDZzMzeN3/f8QGbqn7Tr8XCN+KqJcoaGlN1slJUrfc1ybTuX7i5jOXLkUrs3z1iZM3YIUbNTRdWyh9Fwk4Xv9Kr42cPTh6je4+k3+i814svV5VSrYNKocEpKzCTuyZL8vtp0LkUoYzly5FK7NUtakBGvVOy6/frrryv+/9NPP1kcmzdvniwGpOadKlpV4+Hpw9gXvmH0c18z4NFZhDWNothwk+Liohrx5epyPjdJT9uW/ixff42U69IffVSbzqUIZSxn37F8/vbvy3z45RVWJaRzs7DU4Vzqmmr7Nqll2RR2a+G///1vxf+/+uori2M7d+6UxacUm0UAACAASURBVECRwYSnh20bzmSn1nUte6jVGpq27k2ze/vS4b7HeHjmcvKyrrF2/jSKi4td7suV5Zw6NpyYQQ1JSExndbxjF3xqy7kUpYzlZGSXcCy5gH3H8vk+PoNPl13n8TENeXiI9AdM1FTbt4laLX0TALsubr8f5M57Q+S6V0TU7FRRtRxBrdbQpstIMq79wcGDB13uy1XlnDw6nEkx4WzakcFnSx2/pbY2nEuRylgZzuRSuzfPWCN9EwC7nfHtT0298wmqcj1RVdTsVFG1HKWkuOyrbnZ2tst9uaKck0eHM2VMOL/szOSTJZcdfr/cvupDGe3haC61W7Ok69LIuDy1rXPnzpw+fZro6OiKn//44w9ZDIianSqqli2KCnIwlVin6BUbCkjauwaVSk2HDh1c7ktuvUkxYUwZE86W3ZnMW3zJ6ZsDRD6XIpZRzlxqJc9YOm5PbRM1O1UErbi4OPZtvwqULVkzmYrZt+kLAHQhEdzbrexGjpSzB9i6+m/c03EwQQ3vwtPbn9yMFE4d3EB+9nV6DJ2JXq8HzrqsjHLrjRwYytSxjUlNN3IkKY/+vYItjmfnFHM4SdqoSoRzWZvKKGcutVvzjGvZagq3L20DcbNT3a1lKxt5z3//DYD+nu4VnXFoRCuaR/Un5ex+kg9tpMRYhI9/EGFN2zPw0Vk0b/dAjZRRTr1Wzf0ACAv14o1n7rI6fuxUPoeTzlrtd7UvObVELaOcudRy+nIYF414v/jiC9atW0dWVhYeHh5ERUXx+uuvc++99wJw8uRJZs+ezalTpwgODubJJ59k6tSpVdutydS2ykJcHMVeiEtd01rwc7WlACoNCnIGe0FBzmjZCgpyBntBQY7iinMpRzntBQU5g62gIGepLCjIGeQICircah0RUBm+A6ZIfu358+cJCQkhMDAQo9HIihUrWLJkCTt27KCgoIDBgwczceJEpk+fzqlTp3jmmWeYPXs2Q4cOtaurRGgqKCjUSQoTV0p+rW//SU79DqPRyHfffcdHH33E/v37+fXXX5k3bx47duyoeEDEnDlzOHHiBMuXL7erJcQ0hYLrOX/+PD/++CO7d+/m0qVLGAwGmjZtytChQ5k2bRp+fn4Vr12yZAmJiYmcP3+e7OxsgoKCaN68OVOnTmXQoEFuLIWCggM4MGecm5tLbq51hotOp0On01nt37ZtG6+//jp5eXmoVCqeeOIJAgMDSU5Opm3bthZP6omKiiI2NrZKD0LkGb88/VbE3t5DmWURew9FcCwpx2bEnr2vkIqW7UjC0f0OsGLFCrr36EnPXveh0Xhw4vgxPv30U+I2/Mjcef/G29sbgN9+20uALpARD8WgC9SRn5fHrp07mDlzJpMmT+Vvf31H1nM57tXzNrVuZ8qoEAb1CsDXR83PO3NZvC7D6jWxnzS3ORXw5LjGFVGVp/4sqIiq7NcjmHOXCm1GVdqbDqhL7aKyKScRtKqL2YE542XLljF//nyr/TNnzuTFF1+02v/AAw9w8OBBsrOziYuLo3HjxgDk5+cTEGCZt6HT6cjPr/pCrNtHxqLGG9Y1rSFDhvDg4OH4+/tX7Bs+4iEiIiJYvfp7fvnlZ0aOLHtY6ptvvWP1/pjRY3n5pRmsWxvLO2/bzrpwVVxi8yZejOirY8XGTKaNbuDw+0WNqlS0XByh6cDIeNq0aYwZM8Zqv61R8e0EBQUxdepUunXrRosWLdBqtWRkWA4UcnNz0Wq1VXpw+9oPUeMN65pW+/btLTricu7v+wAAFy9csPt7NRoNDRo0oKioiJIS21fTXRGXqFbBc4+GcjS5kH3HK89ztoeoUZWKlqsjNKWvM9bpdDRp0sRqq6ozBigtLaWkpISLFy/Spk0bTp48SWnprfaWlJREmzZtqtRxe2csarxhfdACSE8vWzMaHBxsdSwvL5ecnGwuX7rE99+t4PChg7Tv0LFiOsPV3gBG9NMR0ciTxWutpyWqi7ujKhUt10ZomjUayZsjLF++nLS0ss9NZmYms2bNwsvLi06dOjF48GBMJhMLFizAaDRy/PhxYmNjmTBhQpW6bp+mEDXesD5omUwmVq36Do1GQ78H+lsdf3b6kxUXNTQaDb3v68MLM6znz1zhDaBRiAePDg1mzS/ZpGWV0DBYvuYqQlSlolU7IzT37t3LwoULuXnzJlqtlvbt27N06VJCQ0MBWLRoEbNmzWLhwoUEBwczY8YMhg0bVqWu3dZ99epVIiIi5ClBJUiN2MsvqfqZW4qWY1pff/UlyadOMnXaEzRpYv1Ukbff+RvGYiMZ6Rns2rUDo8FIYUHlt8LK6Q1g+rgG3MgoIX5bjqTXO0J5VOWS2Ksuiaqsze1CVC1HMbuoM/7iiy/sHm/bti2rV692WNeu2xkzZlT839YVRTkQNd6wrmt9u3wp8Rs3MHTYcB4db/srVFT7DkRHd2XQ4CHMmv0Bvn6+vPH6q+Tk2O4c5Szn/V386dDKl6/XpGOSOWFRlKhKRcvFEZq1LJtCcoTm5cvOpUlVhajxhnVZa+WK5axe9R0PDhrCjJkvS/q9AAMHDiIrK5NffvnFZd4APDQwLaYBR04Vkp1nIjzUg/BQDxqGlH2R8/NVEx7qgZ+P4yMfkaIqFS3XRmiaVWrJmwg4HaEpF6LGG9ZVrZUrlvP9dysY+OAgXnr5VYfOq8FY9nW+spGxXOX08lQTGKChSzs/Pn8nsmKbNbNsLWffrlo+fyeSgT2rXi50O6JFVSparo7QrEN5xvYiNKOjo2UxIGq8YV3U+v67FXz/3Qr6D3iQl195zeIuoXKKigopLLSeFzaZTCTEbwSgU6dOsnu7HYOxlLlLU622r2PL1gIfOVXA3KWpHEySvtRNxKhKRcu1EZpmlUryJgJKhGY90Vq5ciUrVyynYcNGdOrcme3btlocDwoKpnN0F65eucpf3nyN+/rcj14fSUBAABkZ6ezYnkhKSgoDHxxE165dAes7reQqp6kU9h6z7mjLV1OkppfYPF4ZokZVKlpKhObtuH1pG4gZb1jXtE6cOAFAWtoNPpk7x+p4VPsOdI7uQoPQUPoPeJCkpN/Z89tvFBYW4O/vT4sWdzN+wiQeeGBAjZVTLkSNqlS0XNsmzIgx4pWKEqFZD7TK9c78eVEWrZZ33yWrLynZFFKoLJvCGeSMqhS9XYiqVV2yj2yt+kX/I6iz/UFGTSDEyFhBQUFBdmrZNIWSZ6ygoFAnyTy+U/JrQzrc70In0hAiQlOlgnGj9MQMjSC8kQ/ZOUYSd6WxaOUFmwvC7X1VU7RsL6Cv7Cuks1ovvfslVy6c5MqFJLLSUggKjeDNT361+fpyDu/awP6tq7l++Q/MZjPBoRGMf3g43/8cZfP1ztSZyG3szikUfZg3A3sHEx0VQONG3nh5qrh2w8jOA9ms25SGwWitZW/6xNlz2f/RfVb7J46OoGVzP1q18CcizIfrNwxMmGl/Cijxhx6y+qo2gqySkIoQ0xQvPX0r73TV+stleacj9bRsoa0071TRcq/WpthP8NUGor+rLUUFeVX+njVfv8PhnXFEdRtMp94jUanVZKWlcPXqVcB2ZyxCOV2pNaRvSEXO8tY9WRU5y48/0pi+3YNs5izXVBmnT4wkJ6+YM+cL0PpXr5uQ05cjiHIzh1Tc3hmLmp2qaNnXemPuL4Q0Ksuz+PQvIzEYKl9qdmDbGg7tWMe4Zz8muk+MxbGx3dU2R1OilNOVWqLmLANMnHmUazfKbvJZ8q/2+Po4d2OEO/OMa9tqCrf/6RA1O1XRsq9V3hFXhdlsZtvGr4lo1raiIzYU3qSqSxWilNOVWqLmLAMVHXF1cWeecW27HdrtI2NRs1MVLXmyZtOunSfzxiV6DZrEr3FfsHvTcgrzc/D21dKx1wiGtHvT5d5E1aoMd+csy4lbfdWyOWO7fxK2bNnCypW3nrA6btw4Bg4cyMCBA/n5Z3meIV9V3mlwoBceNkJGFC2xtCoj/VrZGuLje39i109LeeChZ5j00r+J6jqI/VtX89xzz9kcJYtaTlfXmatyluU4l87gTl+lKo3kTQTsjowXLVrEJ598UvGz0WhkzZo1FBYW8tZbbzF06NBqGxA1O1XRkidr1lB0E4CbeZk89eZi7onqDUBUt8GYzWb274pDp++Bt397l3kTVcsWIuQsy0ldzDN2FXbdFhcXVzz1FKBLly4EBwcTERFhM0zGGUTNTlW05Mma9fQqm/fUBYdVdMTlRN8/GoDigj9c6k1UrTsRJWdZTtzpy4xK8iYCdjvj8kfulPO3v/2t4v+ZmZmyGBA1O1XRkidrVhccBkBAUKjVsYCghgCYS61XYohaTlfVmUg5y3Ki5BlLx66LDh068MMPP1jtX7VqFR06dJDFgKjZqYqWPFmz4ZGt8PD0JjfTOioxN/M6AGqN9UUcUcvpijoTLWdZTtzpq7ZFaNrtjN9++23WrVvHlClT+Pjjj/n444+ZMmUK69ev5+2335bFgKjZqYqWPFmzXt6+RHUbTF5OOkkHN1sc2/vrqrLX3DFfLLc3UbVAzJxlOXGnrzp1Aa9BgwasWrWKPXv2cPZsWcRgv3796NWrl2wGRM1OVbTsax3etYHs9KsA3MzLoqSkmK1xCwAICo2wuLljyLhXOJu0h1VfvEHvQZMIaqjn9LEdnD66ndGjR7P75N3CltOVWqLmLAMMuj+UsIZeABXTDJPHlj2cODXNyOad0m5GcWeesShzwVKRtM64V69esnbAdyJqdqqiVfn7Dm5fy/nkAxb7Nq/9DIDmbbpZdMZBoRG88N4qNsV+yqGd6ykqyCckLJLhE/4fH/3tCfrG2A50EaGcrtQSOWd5+ICGdGpnObXw1GNlN/ocTcqV3BnL7csRRJkLloqSZ1wPtMr15NRat1+eK+CV3Q7tDK7I05XzXMqRtSxnzjJUHhTkDJUFBTmDHEFBF85ar9KpjGb3tKr276suSoSmgoJCneT8n9K/VTS/+x4XOpGG22+HVlBQUHAFdXLOWC4q+zqU+EOPiri+Vi38KSgw2c1OTfyhB2D9te/JcY0rIglP/VlQEUnYr0cw5y4V2owktPe17+Xpt6L/9h7KLIv+eyiCY0k5NqP/yr9a2Sqns2WUw1e5t8nvXLX5u1Z8EEHyeQOJByzX+5aUmtl3osjm6yv7OupMnSV4trbYd+8//x9Npz+GR4A/F75YQdLL/6g4pp84yubvbfnXmfjfcxdPvX6cc5esb0gSoY3V13bhrK/qUur+HDSHEGJkLFdcnxJJ6Hwk4Y0sE7uPVe+uSjm86Tq3pdlL00h+aw5t57xldfzKdz9a7fPRh9FxyccANjtiELONyelL1Hbh3gjN2tUZC+FWrrg+JZKwepGEGg14ezn/1a7a3tRqOnz5D9I27eT6+s32X3sbTaY9jEpjvxMTsY3J6UvUduHWCM1adju0ECNjV6NEElZN93Y+3NexMRqNipx8E/tOFBK7OY9Cg/Tru9X11uLlx/Fv3YJDj77kkPfIaWMpyb+Jh9bfoffJiTNtTE5EbRfu/ByJ0slKRYiRsStRIgmr5uxlI+u25vHZ91l8GZvFyXNGBvfS8tdnQh0aEVXHm2+zJrR870XOvP8FhRev2HyNLRr074lfi0iurZUn0tUZnG1jciJqu3Dn50gZGQuGEklYNX//0nKOc9fRQi5fL+bRwTqG9Pbnx23S7gKrjrf2//k7Becvc/7TbyT7Boh8chwAl79ZS+S0hx16r1w428bkRNR24dYITUE6WanY7Yz/8Y9/oLITovHuu+/KbkhOqhtJGOzjZfOYuyMJa8JXws58xgwIoFMrH8mdsbPe9BNHEfrgfewZMBlzifSv+Z7BgYSPHkT+qT/J2n1I8vvkpDptTE5EbRfu/ByVmmvXF3+7nXFU1K2n9n7++ee8+OKLLjckF3JEEjaL9MfTQ2X1FcvdkYQ14ctUClm5JgL8pTdoZ73dO+cv3PhpO4brafjd3RQAn4iy6E2PwAD87m6KMT2LkhzLp1DrJ45E4+PN5W/WOFo8WahuG5MTUduFOz9HtW1kbLdGx4wZU7HpdDqLn8eMGVNTHh1GiSSsPp4eEBKoITdf+tdHZ715N2pA2Ij+9E/eXLH12roCgCaTYuifvJmm/5uOuJ0mTzxCqdFIyoo4B0omD3K0MTkRtV24NUKzls0ZSx722JuuEAklktAxX1pf2+f1kQd1eGhUHE6WPgfqrLdD41+y2k7M/DsAN37ewaHxL5Eav9XiPYFdogjseC+p8YkY0+R50IFU5GpjciJqu3Dn58hsVkneRECIC3hyxfUpkYSO+4rpH8A9kV6cOmcgPceEj5eKjq18aHe3N2cvGfllj/SRi7Perq/bZLXP966yD2/BuUs2j0c+8QgAl5dIm6IQsY3J6UvUduHOCM1SQUa8UrHbGXfu3LliRFxUVER0dDQAZrMZlUrF4cOHZTEhV1yfEknouK9T543oG3nQJ9oPra+aUrOZ1AwTP/ySy0+78yl2cNlsTcQlqn28iRg/gsJLV0n7xXb85p2I2sbqQ7twV4RmnbqAd+TIkRox8eqsU7LozF10ibmLLsmiBVBaCqviUlgVV/2LM3KVEeT1dfhUEYdPWecMOItc3govXrHKrKj4HUUGfmnYzSE9UdtYfWgXcvpyBFfNBc+ZM4dt27Zx7do1/Pz86Nu3L2+88QbBwbe+JZ08eZLZs2dz6tQpgoODefLJJ5k6dapd3dr1p0NBQUFBIq6aM9ZoNMyZM4d9+/axYcMGrl+/zltv3cpRyc/P5+mnn6ZPnz7s37+fTz/9lPnz5/Pzz/ZvTFLyjBUUFOokB05nS35t68ZqcnNzrfbrdDp0Op2Nd9wiMTGR1157rWLadt26dcybN48dO3agVpeNd+fMmcOJEydYvnx5pTo1egGvsthFlQrGjdITMzSC8EY+ZOcYSdyVxqKVF2wuCLf3FAZFy/YC+qqiEm1RZCjl6dnXbb5eznP56nzbF4O8PKFvB086t/IgJEBNiQnSskvZk1TMgWTrSctPZmoZ9+p5m1rW2irmvaknrIEnP+/MZfG6DIvjsZ80B8Q7l65oF3Jq2YoJnTg6gpbN/WjVwp+IMB+u3zDYjQmFW1Gh1cGREe+yZcuYP3++1f6ZM2dWeX/Fnj17aNOmTcXPycnJtG3btqIjhrJ7NmJjY+3qCLGa4qWnb+Wdrlp/uSzvdKSeli20leadKlryaJVTWW6to8jlTQU8O9KXZuFqDiSXsOt4MZ4eKqJbejDxQR/Cgo3E7zE67K+c8cOC0fk7F1cp6rkUVWv6xMiKzGatf811OY7c1zdt2jSb905UNSr+73//S2xsLCtWrKjYl5+fT0CAZQCSTqcjP9/+ChS3d8Zy5p0qWrU/z7icpuFqWkRo2H7USNyuW53u7hPFvDXJj95Rnk53xs2beDGir44VGzOZNrqBY+8V9FyKqgXyZTY7iiOrKXS6gCo73jtJSEjg73//OwsWLKBdu3YV+7VaLRkZlt+0cnNz0Wq1dvXcfgFPzrxTRauW5xnfhs//fOTctByCmUohv8iModi5Sx1qFTz3aChHkwvZd7yg6jfcgajnUlQtkC+z2VFcedNHbGwss2bN4ssvv6Rnz54Wx9q0acPJkycpLb01Nk9KSrKYyrCF2ztjOfNOFa3q5dYuea8xi99rzH/eCmPqQzp8vR1rpHJ6u5RqoqDIzIBoLzrerSFIq6JRkIoRvbyIbKhm037nRsUj+umIaOTJ4rUZVb/YBqKeS1G13Imrbodevnw5//rXv1i8eDFdunSxOj548GBMJhMLFizAaDRy/PhxYmNjmTBhgl1dt09TVJV32qFtIB4eKklhIoqWY1rlnL1sZP/vhaRmmPD1VtGxtQ+De2lp09ybWQvTMRilacnprdAAixMKGT/Ah8eH+VbsLzKa+eanIn4/73jkYqMQDx4dGsyaX7JJyyqhYbDjzV/Ucymqljtx4pKHJD744AM8PDys1g0nJCQQERGBVqtl0aJFzJo1i4ULFxIcHMyMGTMYNmyYXV23d8Zy5p0qWrU3z9gWhmK4llnK7+dLuHDdhJ+3ij7tPZky2IfF/y3ij8uOdcjTxzXgRkYJ8dtyHHrf7Yh6LkXVcieuuunj9OnTVb6mbdu2rF692iFdybdD346ct0PLmXeqaNXOPGNbNG6g5qVHfNmw08BvSbeWsR0+U8KbE/wY39+b978tkHxV//4u/nRo5ct7869hqkb1iHouRdVyJ6IEAEnF7bdDy5l3qmjVzjxjW/Tr6ImXh4qjf1quJy4ugZMXS7i/gxchASoycqvW89DAtJgGHDlVSHaeifDQsmYfElj2r5+vmvBQD3LzSykost/JiHouRdVyJ6Za1hm7/QKenHmnilbtzDO2RaC27IOktvHNrHyfWmLr9fJUExigoUs7Pz5/J7JimzWzMQB9u2r5/J1IBva0v/QIxD2Xomq5k9oWoen2zljOvFNFq3bmGdsiNbNshNq9jeWXNx8viGqh4WaRmfQcaaMzg7GUuUtTrbavY8vmyo+cKmDu0lQOJlW91E3Ucymqljsxm6VvIuD2C3hy5p0qWrUzz9gW248V07WNJyN6e9G4gZrz10z4+ajo2c6TQH81a7YVSf4QmUph7zHrjrZ8NUVqeonN464uY33QAvkymx1FlCd4SMXtnTHIm3eqaDmmJWqecVaemU9+KGBIdy9aNtHQuaUHxSa4klbKhl2FnDjnviv5op5LUbXkzGx2BFctbXMVQnTGcuadKlqOIWqeMUBGrpnvtrju7q20rBLJwUK3I+q5FFVLzsxmRygtrV0jYyVCU0FBoU4Sf1j617qHot0/LnW/AwUFQVm4cCFJSUkkJSWRkpKCXq9n69atVq9LSUlh4MCBdrXmzJnDqFGjXGVVwQa1bZhZo52xraxTKMsuLY/Ya9XCn4ICk9280/KsU1s5rC9PvxX9t/dQZln030MRHEvKsRn9Zy8f1lktW+V0ZxnLvblayxk9kbXmzZuHnzYQfbO2+GblUmAw2xxtGYp0THzhY5u/d93SDzAVF9GnTx+GTLM830+Oa8zIgaHsPZLDqT8LMJnMdLxXS78ewZy7VMjLs//AeEcg0qZlnYDa1S6c1aouoixZk4oQI2O5IvbqQ4ygyGUU1ZuzWlu2bOFYWtla5DlvxGAosr3iwtvHjy73j7Taf+GPoxQV5DFkyBBCQkIAy2fn7TyQzar4VAoKb91okpCYwZVUAxNHhTO0XwN+3CLt4pYI9eVqLUepbRfw3L7OGOSL2KsPMYIil1FUb85qRUZGSv4dttiXuBaAcePG2Tx+5kKhRUdczvZ9ZY8Laqb3kfy7RKgvV2s5irLO2I3UhxhBkcsoqjd3nEtD0U2O7f2Z4NAI7rvvPofeGxriCUBWrvQLUKLWlzs/R7XtduhKO+PKQoIAvLy8aNq0Ka+++iq9evVymTlHqQ8xgiKXUVRv7jiXR/f8jKGogH4jnrB4FlpVqFUwaVQ4JSVmEvdkSX6fqPXlzs+RKCNeqVTaGdsLCTKZTJw5c4bXX3+d+Ph4lxhzhvoQIyhyGUX15o5zuS9xLSqVmu4PjHbofc9N0tO2pT9LYq+Scl361Jao9eXOz1Ft64ydmjPWaDS0adOGyZMny+2nWhQZTHh62C6SMzGCcmnJichlFNVbTZ/L6ylnuXjmGK3a9yI41PaTt20xdWw4MYMakpCYzup4xy5qiVpf7vwclZpVkjcRqNYFvMcee0wuH7KQnmmsuPf9TpyJEZRLS05ELqOo3mr6XO5PXAdAjwEPS37P5NHhTIoJZ9OODD5b6vhdb6LWlzs/R7XtAp4Qqynkoj7ECIpcRlG91eS5LCkxcmjXRrS6EKK6DJD0nsmjw5kyJpxfdmbyyZLLTv1eUevLnZ8jU6n0TQTqVGdcH2IERS6jqN5q8lwmHUokPzeTLn1GovHwrPL1k2LCmDImnC27M5m3+JLTozRR68udn6PalmcsxNI2uSL26kOMoMhlFNWbs1pxcXEkHiybMsjPzcJkKmbzui8BCG4YQdf7rW9vdmSKYuTAUKaObUxqupEjSXn07xVscTw7p5jDSdJGjiLUl6u1HEWU6QepCNEZyxmxVx9iBEUto8jenNFau3Yt+/fvt9j3c+znANx9bzerzjgr4xqnj/9Gs1adCNPfXaWnVs39AAgL9eKNZ+6yOn7sVD6Hk85KKR7g/vqqCS1HqG134AnRGcsZsVcfYgRFLaPceu7W+vbbbx1K/gpu0Jh/fXdC8uvnLrrE3EWXqn6hRNxdXzWh5QjKyFhBQUFBAGpbZ6zkGSsoKNRJvt4i/bXTH3SdD6nU6Mi4snhDlQrGjdITMzSC8EY+ZOcYSdyVxqKVF2wuCLcXe+mslq3Yy4mjI2jZ3I9WLfyJCPPh+g2D07GXzvq6M3ZRH+bNwN7BREcF0LiRN16eKq7dMLLzQDbrNqVhMNpep7NpWScrLWf1Ni3rRMzzp21oeTJ+eCgtIr0JCfLAQ6MiLbOYQ7/fZP3mTLJyre+y2rCgtay+KotpdfZcytkudnWMrtin9vEheu0P+DRpwtVVqzn30T8rjjV97lmaPv9spXpy+Sr3JqeWnOeyupQKsmRNKkJMU7z09K2801XrL5flnY7U07KFttK805rQmj4xsiKDWOtfvaqSy9eQviEVGbhb92RVZOA+/khj+nYPspmBW1N6DYI8CQ7UsPdYPhlZJZhKzdwV4c3gPkHc31XHKx9eICdP2m2vcpdTznMph1bTF57DIzjY7mvO/d+/KM7OttjX+sP3XerLFVpyn0up1Lbv/G7vjEXOThUxg1jODFy59Y6fLuD4aevM36SzBbw5Xc+AnoGs35xZ475AvnMph5Z/mzboJ03k/Kf/psXrr1X6uozERAxXr1nss9cZi1TG25H7XEqltnXGbr/pQ+TsVBEziOXMwHWFni3SMspWJGj9pDc3uX3JdS6rraVWwF+9ewAAFkZJREFU0/K9v5L12x4yfrV+hNOdaPz9QSOtIxSmjHdQE23MFqVm6ZsIuH1kXFeyU93ty5kMXFfpeXqo8PFW4+WpIrKxF9PGNATgUNJNt/oSAf3kSfg2b8ap116v8rWdY1fjodViLikh7/ckLn/1tesN1iCuPpeOrU1w/114djvja9eu0bhxY5vHEhMT6d+/f7UN1JXsVHf6cjYD11V6g+4L5NnHwip+Tk03Mm/JVU6eLXSrL3fjrY+g6QvPcWnhVxiuXsM7wvZnqyQvj2tr1pJ39Bglubn4NmtGxOSJtJ3/WQ07dh01cS5NNZtuW23sdsZPPPEEixYtokmTJhb716xZw5dffilLZ1xXslPt4Wpfzmbgukpv37F8rqQa8fFW0yLSm+4dtARonZ9zlMuXu7nn3XcoSknh6rcr7b7u6srvLHds30Fq3Aai1/6AV8OGLnRYc9TEuaxTc8ZvvfUWTz31FBcuXKjYt3DhQpYtW8aKFStkMVBXslPt4Upf1cnAdZVeRnYJx5IL2Hcsn+/jM/h02XUeH9OQh4eEuNWXuwnq2YM/P/gIc4njX8tLcnK4FrvWBa5qnpo6l3Vqzrhfv354eXkxffp0/vOf/xAbG8uJEydYsWIFgYGBshhIzzTSLNIfTw+V1dd4Z7JT5dKSE1f5qm4Grqv1yrl4xcC5FAPD+waxdpO01RQ14aumydq1G2N6Bj7/e9CpV6OyUa6HVotPZCTF2VmY8ioPBjJcvVojPl1JTZ7LOjUyBujVqxcfffQRU6ZM4fLlyyxbtky2jhjqTnZqTfuSIwPXlXp34uWpQuvv+FSFq33VJCF976dr/IaKrcOSRQA0emgEXeM3ED5mjN33+zRtWhM2XUZNn0tzqVnyJgJ2R8blDyU1m80UFxezd+9eevXqhdlsRqVScfjw4Wob+HVnGlPGNeXRGD3HT+ZU7Hc2O1UuLTmR25dcGbhy6wXpNGTbuMuufStfmkZ48/sf1muQa8KXKJx67Q2Lnz2Dg7nn3bfJ3LWb1PVx3DxzBjQaNL6+mPIt/0B7hYXR+NFHatKurLjjXIoSGi8Vu52xvYeSyoXI2akiZhDLmYErt97zE8IIDvTg+OkC0jKK8fRUcU9TH/p01VFYVMo3a9Pc4gvkO5fV0crY8qvFz+WrKYpSUiqOaQK0dPtvPBmJ2yg4d/5/qynuInzMGDR+vsKX0RZyn0uplAoy4pWK29cZg7jZqSJmEMudgSun3o4DefTvqaN/dx26AA1mM6RlFrNpZzbrN2eSniX9wpXc5ZTzXMqpdSelRQbSt/xKQPsoGvR/ALWvLyXZ2WTv20fK0mV0WvltjfiSU0vucykVV42+ExISWLlyJcnJydy8eZPTpy1zWk6ePMns2bM5deoUwcHBPPnkk0ydOrVKXSE6Y1GzU0XMIJY7A1dOvd2H89h9OE8WLbnLKee5lEvLcPWaRXgQgLm4mLOz/uGUnohlBPnPpVRc1RnrdDomTpxIUVER77zzjsWx/Px8nn76aSZOnMiyZcs4deoUzzzzDI0aNWLo0KF2dZUITQUFhTrJP76X/k3srxMcH5fu27ePqVOnWoyM161bx7x589ixYwdqddn6iDlz5nDixAmWL19uV0+IkbGCgoJ0Fi5cSFJSEklJSaSkpKDX69m61X7ORVxcHKtWreKPP/7AbDaj1+sZNmwYM2bMqCHXNY/ZgQt4ubm55ObmWu3X6XTodDob77BNcnIybdu2reiIAaKiooiNja3yvULkGb88/Va85N5DmWXxkg9FcCwpx2a8pL08Y0XLdhznro39XK7lbDlt5ekm/tCjIsKxVQt/CgpMkvJ060MbmzdvHr7aQPR3tcU3K4+bBjPr9lfe86z5+h0O74wjqttgHnxkJCq1mqy0FH47doUZMvkq9yanVnUxmaR/6V+2bBnz58+32j9z5kxefPFFyTr5+fkEBFhmzeh0OvLzq75A6faRsagRmoqW45GjcurJGeEoap05q7VlyxYOXNMD8OlfRmIwVL5k8MC2NRzasY5xz35MdJ+YWlNGOXBkBnbatGmMsbHO25FRMYBWqyUjI8NiX25uLlqttsr3KhGaipZskaNy6skZ4ShqnTmrFfm/O/iqwmw2s23j10Q0a1vRERsKb1bZSYlQRjlw5HZonU5HkyZNrDZHO+M2bdpw8uRJSm97zEhSUhJt2rSp8r1u74xFjdBUtByP9qwPEaaiatki7dp5Mm9c4q6Wnfk17gtmP9+Tvz/TlVnPdmf9N3/HUGQ70rQ2ldEerroDz2QyYTAYKC4uBsBgMGAwGCgtLWXw4MGYTCYWLFiA0Wjk+PHjxMbGMmHChCp13d4ZVxUvGRzohYeHtKxRRct9Wq7QkwtR68zV9ZV+7TwAx/f+xK6flvLAQ88w6aV/E9V1EPu3rmb5vBdsjpJrUxntYTZL3xxhw4YNdOjQgaeeegqADh060KFDBw4cOIBWq2XRokXs2LGDrl278uKLLzJjxgyGDRtWpa7Tc8ZLly7l8ccfd/btFYgaoaloOR7tWR8iTEXVskX5yPdmXiZPvbmYe6J6AxDVbTBms5nDu+LYsWOHS325s0246g68sWPHMnbs2EqPt23bltWrVzus6/TIeOnSpc6+1QJRIzQVLcejPetDhKmoWrbw9Cp7nJEuOKyiIy4n+v7RAOzfv9+lvtzZJkpNZsmbCDjdGct1r0h6prHi3vc7cSZCU9Fyj5Yr9ORC1DpzdX3pgsuethIQFGp1LCCoLL7T1tra2lRGe5SazZI3EXC6M1ap5JnnETVCU9FyPNqzPkSYiqpli/DIVnh4epObaZ0KmJt5HYCQEOvA/9pURnuYzWbJmwjY7Yw7d+5MdHS01da5c2du3JBnbeCvO9MoLTXzaIzeYr+zEZqKlnu0XKEnF6LWmavry8vbl6hug8nLSSfp4GaLY3t/XQWUPUDClb7c2SZKS82SNxFQIjQVLdkiR+XUkzPCUdQ6c1YrLi6OrfvL1u3ezMuipKSYrXELAAgKjbC4uWPIuFc4m7SHVV+8Qe9BkwhqqOf0sR2cPrqdzn1iiI6OBizvmhOhjHIgyIBXMm6/Aw/EjdBUtBzTklNP7phKUevMGa21a9daXXjbvLbsydHN23Sz6IyDQiN44b1VbIr9lEM711NUkE9IWCTDJ/w/7hs6TdgyyoGplqXLC9EZixqhqWi5T0/OCEcQt86c0fr222/tZlHcSXBDPY+9MMflvmpCyxFEeZySVITojBUUFBTkprZ1xkqesYKCQp3k+TnZkl+74I0gFzqRRo2OjG1FJQJMHB1By+Z+tGrhT0SYD9dvGOzGJSb+0AOwHf2nUsG4UXpihkYQ3siH7BwjibvSWLTygs3F5eVRfZPfsX4M+ooPImz+/iJDKU/Pvl7p64dMs/SuD/NmYO9goqMCaNzIGy9PFdduGNl5IJt1m9IwGK19bVrWCbBdZ47WF1QeL+lofYH9CE1n6t/VZXTWF8hT/+Xtddyr5+36h7IlX/Pe1BPWwJOfd+ayeJ1lAljsJ80Bedo+wLefNeWrpas4fmQ/qdeuUlxsICxcT88+AxgW8yg+Preeu3f29El2bdvEubOnuXj+LIaiQp575W0eeHAEAJ1bhsp6LqtLbRsZCzFNMX1iZEV2rda/epZeevpWduqq9ZfLslNH6mnZQltpdqo9ks8bSDxgGVFY4uBJHtI3hJEDQ9l7JIete7Iwmcx0vFfL4480pm/3IF6e/QfGYumaIteXXHpyllFOX67wdjvjhwWj83cuLtSZMq5du5b/blhNl+596PPAYDQaD5JOHGb1t1+xZ+dW3p/7FV7e3gAcObiHTQnr0Ddpyl3N7+GPUyck+XJlfdmjtn3pF6Izliu71hXZqTeyTOw+VuiUn3J2HshmVXwqBYW3RicJiRlcSTUwcVQ4Q/s14Mct0lcHiFpf9SHPWG5vFj6beDGir44VGzOZNrqBY+91soxDhgyh14OP4Od/K2930PAxrI6IZP3qZWz9ZSNDRz5SsX/kwxPx8fFl765EyZ2xq+qrKmrbagq3p7aBfNm1rspO1WjA28v5Ow7PXCi06IjL2b6vbE6rmd7HIT1R66s+5BnL7a0ctQqeezSUo8mF7DteeVh8ZThbxvbt21t0xOX0un8gACkXz1XsCwoOsZi2kIor6ksKrorQdBVCjIzlwhXZqd3b+XBfx8ZoNCpy8k3sO1FI7OY8Cg3VP4GhIZ4AZOVKf3CinCh5xu71dTsj+umIaOTJv75x7iYIucuYkV7mIzDY+nbp2oIonaxUhBgZy4Xc2alnLxtZtzWPz77P4svYLE6eMzK4l5a/PhNarZEylI2EJo0Kp6TETOKerGppOYuSZ+xeX+U0CvHg0aHBrPklm7Qs5/4wy1nGUpOJdauWotFouK/fIKf8iEBtCwqyOzJ+7rnn7L75yy+/lNVMdZE7O/XvX1rO4+46Wsjl68U8OljHkN7+/LjN+ZCT5ybpadvSnyWxV0m57p6vcUqesXt9lTN9XANuZJQQvy3HaQ05y7js639zJvl3Hpv6LBFN7nLak7upbSNju53x0aNHady4MSNGjKBjx47CX50sMpgI9vGyeUyu7NSEnfmMGRBAp1Y+TnfGU8eGEzOoIQmJ6ayOd929+VUhd33VRP07g6i+AO7v4k+HVr68N/8a1bneJFcZV3/7FZvi1zJwaAyjH53qvCEBEL2/uhO7nfHu3bvZvXs3CQkJxMfH069fPx566CFatmxZU/4cIj3TSLNIfzw9VFZf1+TKTjWVQlauiQB/52Z4Jo8OZ1JMOJt2ZPDZ0pq9PfRO5K6vmqh/ZxDVl4cGpsU04MipQrLzTISHln0cQwLL/vXzVf//9s42pqkrDuNPAVFeHYqiNkamGwtTmVLULSLToXPzBWWbaNzQxDFjFOcXaxZNlsxsCYQpJpgRI6iLdDqwEJFsBI3EOHxhUmXa4IYbjhEngqAtK6Xa231o6AZ9ob09cA/0/0vOl3vah+fcwp/bc859LiZFBkDXJcBgdF1IWYyxRFWIsu+/xeKlK5GxQ+nFyPjA7OSbAq+4rCj+/v5ISkpCdnY2iouLMW3aNKSnp6OoqGio/HnEUGSnjgoAxo31h67L86+0H62dhPTUSai63IHcY3957cVbKM9YWl+Bo/wwNswfipnByNs31da+yJwMAEhKCEXevqlIfn3gx7x7O8YSVSHUp44hKfldbP30M2Z55VIyovKMAcBkMqGqqgq7d++GSqVCeno6li3jc1KfZXZqaJDjX8YPloYjwF8GzV3P5nk/XBOF9NRJuFDTgYOFzVzE+1GesbS+ekwCDpxotWtHS6xrFTcbDDhwohU3tANvdfNmjOpTx6A+dQyLlryDbbv2ws9vZKzrWwTB7cYDLqcp9uzZg8bGRiQlJSEzMxMxMTGDYoJVdi3L7NQ1S8Lw0tRANPzRg/anZowJlOG1mDGYOWM07jWbUHXV/aup1cmR2PTeZLS2m3BTq8eSNyL69D95+gwarft6PJ4v1nq85hmz9GYWgGv19oV2QoT1z7K1/bnDfkeIHaNKpUKJqhCRE6Iwe04Cai71DaEf+0IE4ubOBwC0PXqIyxcrAQAtzdZbuzW1NehobwMATNzi+HH0LD9LT+AlNN5dXBbj8vJyBAUF4f79+zh58qTtuMVigUwmg0ajYWKCZXYtq+zUhiYT5BMDkBgfjNAgPwgWC1ofm1FcpcOPNV145sEOpJgXgwEAUZGBUG61X52ub+iCRnvPbT0ezxdrPZ7zjFl7Y4WYMd6+bb2Lrr2tFd/kfmnXHztrrq0YP3r4AMVFR/v01165hNor1pyMVW8vcvgzpDpfvEw/uIvLYnz37t0hMcEyu5ZVdqqmwQhNg5GJpwMFzThQ0MxEC+DzfLHW4znPmLW3/rR1PncrVKg/YsaYlZWF9R/vduu1M+Picbqixmn/3JcjAdgHBQ32+XKGMMwW8EbUHXgEQRC9CJbhVYwpz5ggCGYcOXIEWq0WWq0WLS0tkMvluHjxot3rWlpakJyc7FIrJycHKSkpor2kZja6/dqyw9Jv1x3SK2NnWbO7Pvkv+u9aXYc1+m/VFNRrnzqM/uvNmnWkR1qOoxKdZRCz1BI7TpZazjKzq4sX2GIcY6aHwGAwi87MFvtZOvImpa9ebyy1Dh48iODQsZBHv4qgTh0MPRZUaOwXWHqM4di4PcvhGEtPfAXzMyMSExOdngd3GFF34A0FLOMNSUs6Ld698Ro7yqMvb7T2HqrE+Cjr4lyOcg16jI53g4weEwzFotV2x+//dgtGgx7Lly/HuHHehRQNty/9km8oZBlvSFrSafHujdfYUR59eaPVW4jFcr1aDQBYt26dVzoAIAiC240HJL8yZhn9R1rSafHujRW+4EuqMfYY/0H9tUpERE7BwoULvdYTzNIEP4lF8itjltF/pCWdFu/eWOELvqQa462rlegxGjDvzVQmdwEOt3B5yYuxu9F/pMW3Fu/eWOELvqQa4/VqNWQyP8xfvJaJ3nArxi6nKQ4fPuy0TyaTYceOHV4bYBlvSFrSafHujRW+4EuKMT5suYc/G+vxStxCREQ6fiq7pwy3fcYu/70FBwfbNZlMBrVajYKCAiYG2jtMtvvV++NpvCFpSafFuzdW+IIvKcZYW10KAFjw1vvMNIfblbHLYrxlyxZbS0tLg9FohFqtxooVK3DhwgUmBljGG5KWdFq8e2OFL/ga6jE+f25C3U/nEBo+DrMUbzHTHW6pbQNO/Dx58gS5ublISUmB2WxGWVkZlEolxo/37FHizmAZb0ha0mnx7o0VvuBrqMeoratGl64DisTV8A8YxUxXMJvdbjzgcs44Ozsb58+fR1paGs6dO4eQkBDmBljGG5KWdFq8e+M1dpRHX95o3bhcjs62BwCALl0nzOZnOF9qfVZmxIQpSFhkf3vzYExRACMsQvP48eMIDAxEfn5+n4ePso7QZBlvSFrSafHsjdfYUV59idWqrS7F7w0/9zlWWZIHAJgRO8+uGHc+/hu//nIF0TFzECWf4ZnJARjM6QdBEHDo0CGcOXMG3d3diI+Px/79+yGXywd+sxO4iNBkGW9IWtJpsdbjNfbSF3yJ1dr++QmPXh8xfjK+/u62R+9xl8FcmCsoKEBFRQWKiooQFRWFrKwsbNu2DWfPnhW9R1ryfcYEQRCDgcUiuN085fTp08jIyMD06dMREhICpVKJpqYm1NXVifZLEZoEQYxInKUBOuIH1VzodDq74+Hh4QgP7zuNpNfrkZCQgJKSEsTFxdmOr1y5EuvXr8emTZtE+ZU8m4IgCGIw6I0udYe8vDyHN7llZmZi586dfY51dVm39vUv0mFhYbY+MVAxJgjC59m8eTNSU1PtjvcvuAAQGhoKwHqF/H/0er2tTwxUjAmC8HkcTUc4IywsDHK5HHfu3MHs2bMBWAtxc3MzYmNjRXugBTyCIAgP2bBhAwoLC9HU1ASDwYCcnBxER0dDoVCI1qQrY4IgCA/JyMiAXq/Hxo0b0d3dDYVCgfz8fK+iP2k3BUEQBAfQNAVBEAQHUDEmCILgACrGBEEQHEDFmCAIggOoGBMEQXAAFWOCIAgOoGJMEATBAVSMCYIgOICKMUEQBAf8C2/i1eR/QMKAAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k5JtLTNLeSk8"
      },
      "source": [
        "### Tuning of Hyperparameters\n",
        "\n",
        "We only show below hyperparameter tuning on the best LSTM model (Model 7).\n",
        "\n",
        "We observed that the performance varies a lot. The best results are obtained when the learning rate equals to 0.01 (the one we used before). The `hidden_dim` parameter has only a few impact on the F1-score: as a consequence, we keep it to 50 for coherence with the other models.\n",
        "\n",
        "Note that we reduced here the number of epochs to  the reduce the computational time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mmEZ9_z4eVkX",
        "outputId": "ea5c5d3a-3a44-462c-f8f8-d5f05c9ea2c6"
      },
      "source": [
        "for h_dim in [25,50,75,100]:\n",
        "  for lr in [0.001,0.01,0.05]:\n",
        "    f1 = test_parameter_lstm_w2vec(LSTM_w2vec2_variable_size,\n",
        "                         embedding_dim = w2vecembedding.vector_size,\n",
        "                         hidden_dim = h_dim,\n",
        "                         vec_weights = pretrained_weights,\n",
        "                         freeze_emd = True,\n",
        "                         epochs = 100,\n",
        "                         lr=lr)\n",
        "    print(\"hidden_dim = {}, lr = {}, f1_score = {}\".format(h_dim,lr,f1))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "hidden_dim = 25, lr = 0.001, f1_score = 0.1509343265727008\n",
            "hidden_dim = 25, lr = 0.01, f1_score = 0.3335883043492743\n",
            "hidden_dim = 25, lr = 0.05, f1_score = 0.545553141148038\n",
            "hidden_dim = 50, lr = 0.001, f1_score = 0.15364845782237088\n",
            "hidden_dim = 50, lr = 0.01, f1_score = 0.5204518418461518\n",
            "hidden_dim = 50, lr = 0.05, f1_score = 0.3963331215796773\n",
            "hidden_dim = 75, lr = 0.001, f1_score = 0.24114620979104015\n",
            "hidden_dim = 75, lr = 0.01, f1_score = 0.41355430128958853\n",
            "hidden_dim = 75, lr = 0.05, f1_score = 0.5000884611625779\n",
            "hidden_dim = 100, lr = 0.001, f1_score = 0.21608377102586263\n",
            "hidden_dim = 100, lr = 0.01, f1_score = 0.4187210395562743\n",
            "hidden_dim = 100, lr = 0.05, f1_score = 0.469446446553062\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wbkxA24cPl-v"
      },
      "source": [
        "# 4 Model 8 : Fine-tuning the pre-trained CamemBERT transformer\n",
        "\n",
        "For our last models, we use the famous bidirectional transformer BERT. The pre-trained CamemBERT (French equivalent) model can be easily fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, including sequence classification. For that, we use the `BertForSequenceClassification` function which uses the model transformer with a linear layer on top of the pooled output.\n",
        "\n",
        "For **Model 8**, we used the same training and tests sets as previous models.\n",
        "\n",
        "**Bibliography used for CamemBERT** \n",
        "\n",
        "https://towardsdatascience.com/multi-class-text-classification-with-deep-learning-using-bert-b59ca2f5c613"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5ni5GcA_yP4_",
        "outputId": "748adbf6-2a48-404d-b0a1-b9149af66d8f"
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Thu Apr 15 17:07:16 2021       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 460.67       Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   52C    P0    27W /  70W |   2346MiB / 15109MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 461
        },
        "id": "MQ2_f7gRPvfe",
        "outputId": "4a2076ff-e5a9-4487-cd55-05679552da09"
      },
      "source": [
        "# We keep only the relevant columns : we want to predict the major code (grand domaine)\n",
        "# using the description\n",
        "y_variable = \"major_code\"\n",
        "data_bert= data[[\"description\", y_variable]].copy()\n",
        "\n",
        "# we change the grades to 0-numbering because it is the classical way to do to perform the models\n",
        "tag_to_ix = {}\n",
        "i=0\n",
        "for code in data_bert[y_variable].unique():\n",
        "  tag_to_ix[code]=i\n",
        "  i += 1\n",
        "print(tag_to_ix)\n",
        "data_bert['label'] = data_bert[y_variable].replace(tag_to_ix)\n",
        "data_bert"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'A': 0, 'B': 1, 'C': 2, 'D': 3, 'E': 4, 'F': 5, 'G': 6, 'H': 7, 'I': 8, 'J': 9, 'K': 10, 'L': 11, 'M': 12, 'N': 13}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>description</th>\n",
              "      <th>major_code</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Réalise des travaux mécanisés agricoles, sylvi...</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Tracteur, moissonneuse-batteuse, ensileuse, ép...</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Depuis la cabine de sa machine, le conducteur ...</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Réalise des opérations de coupe et d''entretie...</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Avec ses cordes de rappel, son harnais, ses mo...</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1242</th>\n",
              "      <td>Surveille et coordonne les mouvements des trai...</td>\n",
              "      <td>N</td>\n",
              "      <td>13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1243</th>\n",
              "      <td>Le technicien de la circulation ferroviaire SN...</td>\n",
              "      <td>N</td>\n",
              "      <td>13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1244</th>\n",
              "      <td>Réalise des opérations de mise en fonction, de...</td>\n",
              "      <td>N</td>\n",
              "      <td>13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1245</th>\n",
              "      <td>Sur les pistes de ski, une panne d'une télécab...</td>\n",
              "      <td>N</td>\n",
              "      <td>13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1246</th>\n",
              "      <td>Réalise des opérations de formation de trains ...</td>\n",
              "      <td>N</td>\n",
              "      <td>13</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1247 rows × 3 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                            description major_code  label\n",
              "0     Réalise des travaux mécanisés agricoles, sylvi...          A      0\n",
              "1     Tracteur, moissonneuse-batteuse, ensileuse, ép...          A      0\n",
              "2     Depuis la cabine de sa machine, le conducteur ...          A      0\n",
              "3     Réalise des opérations de coupe et d''entretie...          A      0\n",
              "4     Avec ses cordes de rappel, son harnais, ses mo...          A      0\n",
              "...                                                 ...        ...    ...\n",
              "1242  Surveille et coordonne les mouvements des trai...          N     13\n",
              "1243  Le technicien de la circulation ferroviaire SN...          N     13\n",
              "1244  Réalise des opérations de mise en fonction, de...          N     13\n",
              "1245  Sur les pistes de ski, une panne d'une télécab...          N     13\n",
              "1246  Réalise des opérations de formation de trains ...          N     13\n",
              "\n",
              "[1247 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 66
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 959
        },
        "id": "2EIcDU-0QMZe",
        "outputId": "ff063572-543f-43d1-b0dc-de94c90df7a7"
      },
      "source": [
        "# create train 70 %\n",
        "# create test 30 %\n",
        "X_train, X_val, y_train, y_val = train_test_split(data_bert.index.values, \n",
        "                                                  data_bert.label.values, \n",
        "                                                  test_size=0.3, \n",
        "                                                  random_state=seed_val, \n",
        "                                                  stratify=data_bert.label.values)\n",
        "\n",
        "data_bert['data_type'] = 'not_set'\n",
        "data_bert.loc[X_train, 'data_type'] = 'train'\n",
        "data_bert.loc[X_val, 'data_type'] = 'val'\n",
        "\n",
        "data_bert.groupby(['major_code', 'label', 'data_type']).count()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>description</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>major_code</th>\n",
              "      <th>label</th>\n",
              "      <th>data_type</th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">A</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">0</th>\n",
              "      <th>train</th>\n",
              "      <td>52</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>22</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">B</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">1</th>\n",
              "      <th>train</th>\n",
              "      <td>28</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">C</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">2</th>\n",
              "      <th>train</th>\n",
              "      <td>31</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">D</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
              "      <th>train</th>\n",
              "      <td>52</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>22</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">E</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">4</th>\n",
              "      <th>train</th>\n",
              "      <td>46</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>20</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">F</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">5</th>\n",
              "      <th>train</th>\n",
              "      <td>67</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>29</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">G</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">6</th>\n",
              "      <th>train</th>\n",
              "      <td>45</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">H</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">7</th>\n",
              "      <th>train</th>\n",
              "      <td>151</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>65</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">I</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">8</th>\n",
              "      <th>train</th>\n",
              "      <td>41</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>18</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">J</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">9</th>\n",
              "      <th>train</th>\n",
              "      <td>49</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">K</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">10</th>\n",
              "      <th>train</th>\n",
              "      <td>142</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>61</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">L</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">11</th>\n",
              "      <th>train</th>\n",
              "      <td>38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>17</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">M</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">12</th>\n",
              "      <th>train</th>\n",
              "      <td>85</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>37</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">N</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">13</th>\n",
              "      <th>train</th>\n",
              "      <td>45</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                            description\n",
              "major_code label data_type             \n",
              "A          0     train               52\n",
              "                 val                 22\n",
              "B          1     train               28\n",
              "                 val                 12\n",
              "C          2     train               31\n",
              "                 val                 13\n",
              "D          3     train               52\n",
              "                 val                 22\n",
              "E          4     train               46\n",
              "                 val                 20\n",
              "F          5     train               67\n",
              "                 val                 29\n",
              "G          6     train               45\n",
              "                 val                 19\n",
              "H          7     train              151\n",
              "                 val                 65\n",
              "I          8     train               41\n",
              "                 val                 18\n",
              "J          9     train               49\n",
              "                 val                 21\n",
              "K          10    train              142\n",
              "                 val                 61\n",
              "L          11    train               38\n",
              "                 val                 17\n",
              "M          12    train               85\n",
              "                 val                 37\n",
              "N          13    train               45\n",
              "                 val                 19"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w-n0tSz-_xGs"
      },
      "source": [
        "In the following code, we tidy the dataset so that it is easily incorporated in the model.\n",
        "\n",
        "In particular, we specify a maximum length for the sentence (= maximum number of tokens) of 256. Indeed, we noticed that the increase of this value dramatically increases the computation time without any significant gain in the quality of the model. In other words, the first tokens seem to be relevant enough to give a good prediction of the labels. This value is higher than the one used in the previous section because CamemBERT divides the sentence in much more tokens (especially due to the byte pair encoding for unknown words)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c4KTsIAtRNGe",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35,
          "referenced_widgets": [
            "72b3b3a25ab4452ebee962bf0f627b0a",
            "f41cc55d32974a788b6aabb78f9f8eac",
            "80d6c9bafe3b42bc94b9a29715e1e3c0",
            "a59ce2ff4f284bc09602866cf243c4bf"
          ]
        },
        "outputId": "3767cf8a-b1a7-441b-e549-0b5e7f00ceb9"
      },
      "source": [
        "tokenizer = CamembertTokenizer.from_pretrained(\"camembert-base\", \n",
        "                                          do_lower_case=True)\n",
        "                                          \n",
        "encoded_data_train = tokenizer.batch_encode_plus(\n",
        "    data_bert[data_bert.data_type=='train'].description.values, \n",
        "    add_special_tokens=True, \n",
        "    return_attention_mask=True, \n",
        "    pad_to_max_length=True, \n",
        "    max_length=256, \n",
        "    return_tensors='pt'\n",
        ")\n",
        "\n",
        "encoded_data_val = tokenizer.batch_encode_plus(\n",
        "    data_bert[data_bert.data_type=='val'].description.values, \n",
        "    add_special_tokens=True, \n",
        "    return_attention_mask=True, \n",
        "    pad_to_max_length=True, \n",
        "    max_length=256, \n",
        "    return_tensors='pt'\n",
        ")\n",
        "\n",
        "\n",
        "input_ids_train = encoded_data_train['input_ids']\n",
        "attention_masks_train = encoded_data_train['attention_mask']\n",
        "labels_train = torch.tensor(data_bert[data_bert.data_type=='train'].label.values)\n",
        "\n",
        "input_ids_val = encoded_data_val['input_ids']\n",
        "attention_masks_val = encoded_data_val['attention_mask']\n",
        "labels_val = torch.tensor(data_bert[data_bert.data_type=='val'].label.values)\n",
        "\n",
        "dataset_train = TensorDataset(input_ids_train, attention_masks_train, labels_train)\n",
        "dataset_val = TensorDataset(input_ids_val, attention_masks_val, labels_val)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "72b3b3a25ab4452ebee962bf0f627b0a",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=810912.0, style=ProgressStyle(descripti…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2XPopTES8FlL",
        "outputId": "cad4da03-9475-4754-edfc-6008edbfdebd"
      },
      "source": [
        "# To understand the previous max_length parameter two examples :\n",
        "## - in the first one the number of tokens is lower than max_len (the number 1 is used to complete the sentence)\n",
        "## - in the second one the number of tokens is higher than max_len (truncated sentence)\n",
        "print(tokenizer.tokenize(\"Bonjour, j'espère que vous appréciez la lecture de ce projet !\"))\n",
        "print(tokenizer.batch_encode_plus(\n",
        "    [\"Bonjour, j'espère que vous appréciez la lecture de ce projet !\"], \n",
        "    add_special_tokens=True, \n",
        "    return_attention_mask=True, \n",
        "    pad_to_max_length=True, \n",
        "    max_length=20, \n",
        "    return_tensors='pt'\n",
        ")['input_ids'])\n",
        "print(tokenizer.batch_encode_plus(\n",
        "    [\"Bonjour, j'espère que vous appréciez la lecture de ce projet !\"], \n",
        "    add_special_tokens=True, \n",
        "    return_attention_mask=True, \n",
        "    pad_to_max_length=True, \n",
        "    max_length=10, \n",
        "    return_tensors='pt'\n",
        ")['input_ids'])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['▁bonjour', ',', '▁j', \"'\", 'espère', '▁que', '▁vous', '▁appréciez', '▁la', '▁lecture', '▁de', '▁ce', '▁projet', '▁!']\n",
            "tensor([[    5,  5061,     7,    76,    11,  1612,    27,    39, 25331,    13,\n",
            "          1147,     8,    44,   327,    83,     6,     1,     1,     1,     1]])\n",
            "tensor([[    5,  5061,     7,    76,    11,  1612,    27,    39, 25331,     6]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9-dR03MURl6P",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52,
          "referenced_widgets": [
            "5ba89c16444943d9b0f16353d73cf95c",
            "633004165163472e80dd7ea899248a48",
            "750522277df847ce97c823761f04f474",
            "cd749f2ec41c4358a1b6ab259deb9895",
            "8c29a3e79be140deb1a0ffd98bc0869e",
            "f2a46fc98ad94ca6895ebb1f82a64567",
            "164e25d413804caf97681358bcf76c87",
            "5ab351a64b0042e4b16ff6087a27ea26"
          ]
        },
        "outputId": "a271c534-9f95-480b-bcf7-c25dd24dda3a"
      },
      "source": [
        "#load the pretrained CamemBERT model\n",
        "camembert = CamembertForSequenceClassification.from_pretrained(\"camembert-base\",\n",
        "                                                      num_labels=len(tag_to_ix),\n",
        "                                                      output_attentions=False,\n",
        "                                                      output_hidden_states=False)\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "camembert.to(device)\n",
        "optimizer = AdamW(camembert.parameters(),\n",
        "                  lr=1e-5, \n",
        "                  eps=1e-8)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "5ba89c16444943d9b0f16353d73cf95c",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=508.0, style=ProgressStyle(description_…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "633004165163472e80dd7ea899248a48",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=445032417.0, style=ProgressStyle(descri…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KSLftUhPRw2h"
      },
      "source": [
        "# 5 epochs is enough to have good performance             \n",
        "epochs = 5\n",
        "batch_size = 3\n",
        "\n",
        "dataloader_train = DataLoader(dataset_train, \n",
        "                              sampler=RandomSampler(dataset_train), \n",
        "                              batch_size=batch_size)\n",
        "\n",
        "dataloader_validation = DataLoader(dataset_val, \n",
        "                                   sampler=SequentialSampler(dataset_val), \n",
        "                                   batch_size=batch_size)\n",
        "# check documentation, mentionned here: https://huggingface.co/transformers/training.html\n",
        "scheduler = get_linear_schedule_with_warmup(optimizer, \n",
        "                                            num_warmup_steps=0,\n",
        "                                            num_training_steps=len(dataloader_train)*epochs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QJcszLavR8tL"
      },
      "source": [
        "# To compute the f1-score\n",
        "def f1_score_func(preds, labels):\n",
        "    preds_flat = np.argmax(preds, axis=1).flatten()\n",
        "    labels_flat = labels.flatten()\n",
        "    return f1_score(labels_flat, preds_flat, average='weighted')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WRZQ8AdN8BSF"
      },
      "source": [
        "# The model by epoch is stored in a folder data_volum\n",
        "!mkdir data_volume"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W8zY2ZXNSBQh"
      },
      "source": [
        "# 6 minutes to run\n",
        "random.seed(seed_val)\n",
        "np.random.seed(seed_val)\n",
        "torch.manual_seed(seed_val)\n",
        "torch.cuda.manual_seed_all(seed_val)\n",
        "\n",
        "#function to evaluate the model on the validation dataset\n",
        "def evaluate(dataloader_val):\n",
        "\n",
        "    camembert.eval()\n",
        "    \n",
        "    loss_val_total = 0\n",
        "    predictions, true_vals = [], []\n",
        "    \n",
        "    for batch in dataloader_val:\n",
        "        \n",
        "        batch = tuple(b.to(device) for b in batch)\n",
        "        \n",
        "        inputs = {'input_ids':      batch[0],\n",
        "                  'attention_mask': batch[1],\n",
        "                  'labels':         batch[2],\n",
        "                 }\n",
        "\n",
        "        with torch.no_grad():        \n",
        "            outputs = camembert(**inputs)\n",
        "            \n",
        "        loss = outputs[0]\n",
        "        logits = outputs[1]\n",
        "        loss_val_total += loss.item()\n",
        "\n",
        "        logits = logits.detach().cpu().numpy()\n",
        "        label_ids = inputs['labels'].cpu().numpy()\n",
        "        predictions.append(logits)\n",
        "        true_vals.append(label_ids)\n",
        "    \n",
        "    loss_val_avg = loss_val_total/len(dataloader_val) \n",
        "    \n",
        "    predictions = np.concatenate(predictions, axis=0)\n",
        "    true_vals = np.concatenate(true_vals, axis=0)\n",
        "            \n",
        "    return loss_val_avg, predictions, true_vals\n",
        "    \n",
        "for epoch in tqdm(range(1, epochs+1)):\n",
        "    \n",
        "    camembert.train()\n",
        "    \n",
        "    loss_train_total = 0\n",
        "\n",
        "    progress_bar = tqdm(dataloader_train, desc='Epoch {:1d}'.format(epoch), leave=False, disable=False)\n",
        "    for batch in progress_bar:\n",
        "\n",
        "        camembert.zero_grad()\n",
        "        \n",
        "        batch = tuple(b.to(device) for b in batch)\n",
        "        \n",
        "        inputs = {'input_ids':      batch[0],\n",
        "                  'attention_mask': batch[1],\n",
        "                  'labels':         batch[2],\n",
        "                 }       \n",
        "\n",
        "        outputs = camembert(**inputs)\n",
        "        \n",
        "        loss = outputs[0]\n",
        "        loss_train_total += loss.item()\n",
        "        loss.backward()\n",
        "\n",
        "        torch.nn.utils.clip_grad_norm_(camembert.parameters(), 1.0)\n",
        "\n",
        "        optimizer.step()\n",
        "        scheduler.step()\n",
        "        \n",
        "        progress_bar.set_postfix({'training_loss': '{:.3f}'.format(loss.item()/len(batch))})\n",
        "         \n",
        "        \n",
        "    torch.save(camembert.state_dict(), f'data_volume/finetuned_CAMEMBERT_epoch_{epoch}.model')\n",
        "        \n",
        "    tqdm.write(f'\\nEpoch {epoch}')\n",
        "    \n",
        "    loss_train_avg = loss_train_total/len(dataloader_train)            \n",
        "    tqdm.write(f'Training loss: {loss_train_avg}')\n",
        "    \n",
        "    val_loss, predictions, true_vals = evaluate(dataloader_validation)\n",
        "    val_f1 = f1_score_func(predictions, true_vals)\n",
        "    tqdm.write(f'Validation loss: {val_loss}')\n",
        "    tqdm.write(f'F1 Score (Weighted): {val_f1}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uuV7OnxI69vJ"
      },
      "source": [
        "key_list = list(tag_to_ix.keys())\n",
        "val_list = list(tag_to_ix.values())\n",
        "def matrix_pred_bert(valid_dl):\n",
        "  # We retreive the predictions and the true values\n",
        "  val_loss, predictions, true_vals = evaluate(dataloader_validation)\n",
        "  preds_flat = np.argmax(predictions, axis=1).flatten()\n",
        "  labels_flat = true_vals.flatten()\n",
        "\n",
        "  y_true = [key_list[val_list.index(i)] for i in true_vals]\n",
        "  y_pred = [key_list[val_list.index(i)] for i in preds_flat]\n",
        "  data = {\"description\": data_bert[data_bert.data_type=='val'].description.values,\n",
        "       \"label\": y_true,\n",
        "       \"y_pred\":y_pred}\n",
        "  return pd.DataFrame(data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wK0vsG_b7sC6"
      },
      "source": [
        "pred_vs_true_m8 = matrix_pred_bert(dataloader_validation)\n",
        "y_true = pred_vs_true_m8.label\n",
        "y_pred = pred_vs_true_m8.y_pred"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 647
        },
        "id": "hftc50JqDTWz",
        "outputId": "9455af85-8560-43e7-c17a-9d30b1cba4bd"
      },
      "source": [
        "print( classification_report(y_true, y_pred ) )\n",
        "cm = confusion_matrix(y_true, y_pred , labels = key_list) \n",
        "df_cm = pd.DataFrame(cm, index=key_list, columns=key_list)\n",
        "# config plot sizes\n",
        "sn.set(font_scale=1.2)\n",
        "sn.heatmap(df_cm, annot=True, annot_kws={\"size\": 18}, cmap='coolwarm', linewidth=0.5, fmt=\"\")\n",
        "plt.tick_params(axis='both', which='major', labelsize=10, labelbottom = False, bottom=False, top = False, labeltop=True)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           A       0.94      0.77      0.85        22\n",
            "           B       1.00      0.08      0.15        12\n",
            "           C       1.00      0.77      0.87        13\n",
            "           D       0.74      0.91      0.82        22\n",
            "           E       0.77      0.50      0.61        20\n",
            "           F       0.65      0.83      0.73        29\n",
            "           G       0.94      0.89      0.92        19\n",
            "           H       0.72      0.89      0.79        65\n",
            "           I       0.88      0.39      0.54        18\n",
            "           J       0.89      0.76      0.82        21\n",
            "           K       0.77      0.82      0.79        61\n",
            "           L       0.81      1.00      0.89        17\n",
            "           M       0.68      0.70      0.69        37\n",
            "           N       0.90      0.95      0.92        19\n",
            "\n",
            "    accuracy                           0.78       375\n",
            "   macro avg       0.84      0.73      0.74       375\n",
            "weighted avg       0.80      0.78      0.76       375\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEDCAYAAADgNmW7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd3wUZf7H37ubzrIplEBCaEox9KD0DtK7ICLtVNCj6Nnu/Mlxp6goHkXvRIqCJ0gPXZr0DtJBAqFJCy2VNFJ39/fHXgLrbjYzm9nskzDv12ter+wzM5/9PjNPnn3mKZ/RmM1mMyoqKioqbkXr7gBUVFRUVNTKWEVFRUUI1MpYRUVFRQDUylhFRUVFANTKWEVFRUUA1MpYRUVFRQCEqox37NhBnTp1uHr1apF0nnnmGfr160ffvn0ZMGAAJ0+eLJJeXFwc77zzDl26dGHgwIGMGTOGa9euOR1Xr1696Nu3Lz/88AMmk8npuPL08rbvvvtOMa2YmBinteLj43nvvffo3LkzAwcOZMiQIWzfvl22TpMmTaw+r1mzhk8++cTpuArSdbfe4+fv3buXbt26cfv2bbfFk0edOnV4//338z/n5ubSokUL3njjDae0pk6dmv95wYIFfPPNN4rEWVrwcHcAj7Nx40aaNm3Kpk2beOutt5zW8fHxYf369QDs37+fmTNnsnjxYqe0zGYzEyZMoH///nz11VcAREdHk5CQQI0aNZyOKyEhgffee4+0tDSn8/q4XlFRSstsNjN+/Hj69+/PjBkzALh9+za7du0qsnZp5/Dhw3z22WcsWLCA0NBQd4eDn58fly9fJjMzEx8fHw4ePEhwcLBTWl5eXmzbto3XX3+doKAghSMtHQjTMk5PT+fEiRNMmTKFTZs2KaablpaGwWBw+vwjR47g4eHB0KFD89Pq1q3Ls88+W6S4ypUrx6effsqSJUsoTetujhw5gqenp9X1Cg0NZcSIEW6MSnyOHTvGpEmTmDt3LlWrVnV3OPm0b9+ePXv2ALBp0yZ69erllI6HhwdDhgxh4cKFCkZXuhCmZbxz507atm1LjRo1CAwM5Ny5c9SvX98prczMTPr160dWVhZxcXFFKgCXL1+mXr16Tp/viLCwMIxGIwkJCZQvX172+Xn5zOONN96gZ8+eTsXyuFaVKlX49ttvndK5fPky4eHhTp3rKCaA5ORkOnXqpIi2SGRnZzN+/HgWLVrEU0895e5wrOjZsyezZ8+mY8eOXLx4kRdeeIETJ044pTVs2DD69u3L6NGjFY6ydCBMZbxp0yZGjhwJWArApk2bnK6MH3/kPnXqFB988AEbN25Eo9EoFq8IiNhN8UcmT57MiRMn8PT0ZPXq1UWKac2aNZw7d07pEN2Oh4cHTZo0YdWqVUyaNMnd4VhRt25dYmJi2LhxI+3bty+Sll6vp1+/fixatAgfHx+FIiw9CNFN8eDBA44cOcKkSZPo1KkTCxYsYMuWLYo8vjdp0oSkpCQSExOdOr9WrVpERUUVOQ573Lp1C51OR7ly5Vyi7w5q1arF+fPn8z9/9NFH/PjjjyQlJbkxKrHRarV8/fXX/Pbbb8ydO9fd4djQqVMn/vWvfzndRfE4o0aNYvXq1WRkZCgQWelCiMr4l19+oV+/fuzevZtdu3axd+9eqlSpwvHjx4usffXqVYxGIwEBAU6d36JFC7Kzs1mxYkV+WnR0dJFjS0xM5KOPPmLYsGGlqsXeokULsrKyWLp0aX5aZmamGyMqGfj6+jJv3jx+/vlnIiMj3R2OFYMGDWL8+PHUqVOnyFoBAQF0796dVatWKRBZ6UKIboqNGzcyZswYq7SuXbuyceNGnnvuOdl6j/c1ms1mvvzyS3Q6nVOxaTQaZs2axeeff87333+Pt7c3oaGhTJw40em4cnNz0el09OvXj1deecWpuB7Xy6Nt27ZWU5HcgUaj4dtvv+WLL75g/vz5BAUF4evr6/a4XEVubi5eXl6KaAUEBDB//nyGDRtGUFAQnTt3dkonIyODdu3a5X9+5ZVXilTOKlWqlN+FqASvvvoqS5YsUUyvtKBRLTRVVJwnOjqaSZMmqS09lSIjRMtYRaUksmzZMn766SennpJUVP6I2jJWUVFREQAhBvBUVFRUnnTUylhFRUVFANTKWEVFRUUA1MpYRUVFRQDUylhFRUVFANTKWEVFRUUA1MpYRUVFRQDUylhFRUVFAIp1Bd68efOIiooiKiqKmJgYQkND7b4BIiYmptB1+dOmTaNv376uClVFRaWEs8lTurFRr5yLLoxEGsVaGc+cOROfMgEEh4Xj45dKagb8YOdtPNlZQfT+07/samxf8Smm3EzatGlDmz57bfb/ZcxTDO5bhb2H4jhyIpFqYWUY1DuEM1HJvP2Ps/xxveGBny0eraVZK0+v12hrL+BGdcswpFcFon/PICEpB6PJTPVQH7q0DuBhpok3J18hOdVoo7Vpfn27cTmbzw6DDtvo7FnVkuTUHC7/nk7tmnoeZuTy0rhTdr/z8XOUjAuUvZdK5HPPqpaKxZUXm6haTxrFWhm/8ckOAiqEAbDgk95kZz20e5yXtx/1mvezSb/9+ymyMlLp1q2b3fdo1ajqxwu9Q9lzKI5JXzzy1L17P4N33qhFl3YV2b43VlKsT4LWmeh0zkSn26Sfu5TOh2Or0qV1IKu3xkvSUjq2oeNOcjc2C4D/zmyEr4/zPWqiXn9QLp+i5lHp6yUHjWfJsqYt1j7jvIrYWc4ctPi8Dh482O7+Lu0qotVqWLne+s26P/9yl4xMI107VJT8XU+CVkHEJuYAoPeTZzuqZGx5FZQSiHz9lcqnqHksjvJaEDpfneRNBEqMa1t2ZjoXT2zBEBRK69at7R5Tt1ZZjEYzFy6lWJ+bY+by72nUrVVW8vc9CVp5eHpo8PXR4umpoWplH14ZZHkD8PHfUmXpuCI2JRD9+iuBqHl05/XSepTylvHx48eZPHmyK2JxyIUTm8nOekiDlgPRau2HXT7Ii+SUHHJybTuh4hOzCPT3wkPiDXoStPLo1jaQZV8/w6Jpdfns3eqU8dUx7ftbRF22341UnLEpgejXXwlEzaM7r5fGUyN5EwFJLePz58/z888/88svvxAaGkrXrl1dHZcNZw+uQqPR0rDVwAKP8fHWkZNrsrsvO9v0v2O0pOXaDko9iVp5HD6dQsy9LHy8tTxV1Zfmjcti0Mt/aHJFbEog+vVXAlHz6M7rVdJaxgX+x127do1NmzaxceNGAgMD6dmzJ2azmZ9++qk44wMg/u4V7lw7TY3wNhiCQgo8LjPLSKCP/VfgeHlp/3eM/YLxJGrlkZCUS0JSLgBHTqdy8GQyX//9Kby9NERukT6A54rYlED0668EoubRnddLlBavVArspujRowdHjhxh3rx5LFu2jBEjRhTYPeBqzh60vNKmYWv7A3d5xCdm42/wxNPOL2L5IG+SkrPJtfO49KRqFcT1mCyu3sykV0d5b60ujticoaRdf2cQNY/uvF5aD43kTQQKrF1nzZpFhQoVGDlyJJMmTeLw4cO446Ugxtxson5dj58+iFqNHC8Eib6cik6n4ZnaBqt0L08NtWrquXglTfL3PglajvDy0lC2jLxR5uKKTS4l8frLRdQ8uvN66by0kjcRKDCKLl268NVXX7FlyxaaN2/OwoUL818vf+DAgWIL8PLZXTxMS6Re837odJ4Oj925Pw6TycyL/UKt0vt0q4yvj45te+5L/t4nQSvQYL+XqmGdMlQL9SH6d3kDeErGpiSiXn8lETWP7rxeGq1G8iYChY7S+Pn50adPH/r06UNycjJbt27l+++/p02bNrK/7Nyv60hJuAPAw7REjLk5HNo8GwBDuRDqN+9vc47ULgqA32+ks2bTHQb1CWXKh+EcPpFI9Sp+DOoTyqnfHsiaXP4kaI0bHkKQvwdnotOJTcjGy1PL09V8aNfMn4xMEwtW3pOspXRsz7crT6UK3gD4Gzzw9NAy4gXLP/S9uCy275Pely3q9Vcyn6LmUenrJQeNTowWr1RkDZn7+/szZMgQhgwZ4tSXnT24mluXj1ql7f/53wCE1WpmUxmnJN7l+oWDhNZsQvnKT0n6jv/Mv8K92Ez6dqtMy+fKkZySw6qNd1iw5JrdZZdPstbeow/o3DKATi0D8C+rw2yG2IQctu5NYvUv8cT9b/GHO2Lr1bkijev5W6W9NrQqAKejkmVVxkrGpbSWkvkUNY9KaslBqxOjxSuVYl308fK78mZiGIIq87fZF2SdYzLB8nUxLF8XI+u8J1HrwPEUDhxPKfxAGSgV29sfnS/8IBmIeP1B2XyKmkclteQgSveDVErMCjwVFRUVOYgyMCcVjdkdUyRUVFRUXMyJjvZtE+zRdPdBF0YiDbVlrKKiUipRuykcoLTXrGj+sKL7Gbtay9l8vjE10UZHA7w3rCxPhXpw+Fw2127n4uUJz4V7UyPEg1+OZLBmT4bVOfP+L8humQAxyoWS5dXVHtAilLGiog7gyUT1hy0dWorHFuJBrTBPdhzLJHLno/nOe05mMfl1f9o29rapjB0hYrlQMi4ltUQuY3IoaS1jt/dwq/6wpUNLaT0fb8s/UnKqtW+B0QRpD81kyZx1J2K5UDIuJbVELmNy0Gi1kjcRcHvLWPWHLR1aSutdv5NLeqaJri18iE82cu2uES8PaNnAm2qVdCz5Rd7qQKUQtbwqichlTA46TzEqWam4PVrVH7Z0aCmt9zDLzOxVaaRnmHljQFmmjgvgk9cD6BDhw9y1aRw4o1yLUg6illclEbmMyaGkLYeWXRknJiYqahgk1e+0uFEyridByxV6WTlm7sQb2fZrBnPWpLJocxpxSUZG99XzTHX3PNSJWl6VROQyJoeS1k3hMIrTp08zYsQIJkyYwPnz5+nduzd9+vShVatW7Nu3T5EAMrOMeHrYD8Pd/rBKxfUkaCmtF1JBx9+GG7hwLYfVuzM4fSmHg2ez+dfiFJLTTQzvUQaNGxo0opZXJRG5jMmhVLWMP/nkE9544w169erFqFGj+Oyzzzh48CCLFy9m5syZigSg+sOWDi2l9bo854OXp4YT0dlW6Tm5cO5qDuX9dZTzL/4WjajlVUlELmNyKFWVsdFopE2bNvTo0YPy5cvTuHFjAJ56SpppjxRUf9jSoaW0XoDeUjTtPUHm/e+4w5RL1PKqJCKXMTloPXSSNxFwWJwff7OHj4+P1T6NQs+Iqj9s6dBSWu9uguWdaC0beFul+3praFTLi/QME7FJxd8dIGp5VRKRy5gctDqN5E0EHI6CREdHExERgdlsJisri4iICADMZjPZ2dmOTpWM6g9bOrSU1tt5LJMW9b0Y0MGX0Ao6rsbkUsZXQ5tG3gSU1bL0l3RZ9osilgsl41JSS+QyJgdXdT988803zJ4926qB2rFjx/yu2/Pnz/PJJ59w4cIFAgMDefXVVxk5cmShug4r4wsX5NlXOovqD1s6tJTUS0wx8cXCFHq39qVuNU+ee8aL7FyIic1l1a6HnLokb9WHqOVCybhEzaO7/IxdOUvi2Weftfty5rS0NEaPHs3LL7/MwoULuXDhAq+//joVK1ake/fuDjXdvugDVH/Y0qKltF78AxM/bkpXICpxy4WScYmax5LgZ5ySkkJKiq23t8FgwGAw2DnDPtu2bUOr1TJu3Di0Wi2NGzdm8ODBLF26tGRUxioqKipKI6cyXrhwIbNmzbJJnzBhAm+++aZN+rlz52jRogW+vr5ERETw9ttvExYWRnR0NOHh4VbjbfXr1ycyMrLweFU/YxUVldLIrXEvSD7Wf+p/JbeML126RJkyZQgJCSE2NpYZM2Zw8uRJ1q9fz5QpU8jKymLGjBn5xx88eJAxY8Zw/rzjJxchLDQ1GhjcN5R+3UOoVNGHB8nZ7D4Qx/wl1+1OCHdkL+ms1pkruVSpoKWcv5bEFBNTFhfsCFa1opYezb2oGmz59bt+z8imwzm8N8RX8biU0MrTs2e7+PKAEGrX1FO7ZhlCgn24F5vptO2is/kUVQuUvZdKXH9H1p6ilgtn4yoqcvqM5XRH1K5dO//v4OBgpkyZwrPPPsupU6fQ6/UkJCRYHZ+SkoJery9UV4huirdGP/I7Xb72lsXvtE8otWrqC/Q7VVrr6So6bseZ8PV2/GVVg7WM6+dDcrqZX45aZpS0buDJ+AE+Ds8TIY/2eH1YtXwPXL1f0YuDqPkUVUvJ6/8kxCWLYlqiqdFo0Gg0mM1m6taty5YtWzCZTPldFVFRUdStW7dQHbdXxqJ4p/7zB4sL2PtDfPH2LPg7BrTxwmiCb9dlkpJuKUWnrxr5YKiv8Hm0h5J+uqLmU1QtENODWOS45OCqqW2bN2+mRYsWBAUFkZCQwPTp0wkKCqJJkyYATJ8+nTlz5jBmzBiio6OJjIzk448/LlTX7Q4ZJck7tZxBQ9VgHWeu5uZXxAAp6WbOXM0tlrhE9tMVNZ+iaoGYHsQixyUHVxkFbdiwgZ49e9KoUSP69+9PVlYW//3vf9Hr9ej1eubPn8++fft49tlnefPNNxk/fjw9evQoVNdhy/jGjRvEx8fTtGlTq/QTJ05QoUIFqlatKisT9ihJ3qlVK1pu2o17tv1cN+6ZaP6M6+MS2U9X1HyKqqUkaly2uKplPHfuXIf7w8PDWbFihWxdhz8Jn3/+ud2OZ71ez+effy77y+xRkrxTDWX+9/aJdFt9e2muiEtkP11R8ymqlpKocdlSqrwp4uPjqVOnjk16nTp1uH37tp0z5FOSvFPznKdyjbYFy16aK+IS2U9X1HyKqqUkaly2lCrXttTU1AL3ZWZmKhJASfJOzft197BjLGIvzRVxieynK2o+RdVSEjUuO2i10jcBcBhF/fr1WblypU16ZGQk9erVUySAkuSdmjdo51/GVt9emiviEtlPV9R8iqqlJGpctuRNOZOyiYDDynjixImsWbOGESNGMHXqVKZOncrw4cNZtWoVf//73xUJoCR5p96MtfyCV6tke9nspbkiLpH9dEXNp6haSqLGZUupeu1S+fLlWb58OePHjyc0NJTQ0FDGjx/PihUrqFChgiIBlCTv1IQUMzdjjTR6ygOD36NfU4OfhkZPFTwxpSTlsSiImk9RtZREjcsWjYdO8iYCkhZ9tGjRghYtWrgkAFG8U7s0taz00Ptq0GkffU5KNXPi0qM5xOsPZDO2nw/jB/hw4DeLjWObBp4OF/uIkkd7KOmnK2o+RdUCMT2IRY5LDqIMzEnF7SvwQAzv1B7Nvex+vnLbaFUZX79nYva6THo096JHcy/MZrhxz8iiX3J498WCV+GJkEd7KOmBq3RsT4KWqB7EosYlB41GjO4HqQhRGYvgnfrebOm+uTfum5i7Qd5sEhHyaA8lPXBB3HyKqiWqB7GoccmihLWMVQtNFRWVUsmDLydIPjbgA1sv4+JGiJZxaeLatWts2LCBgwcPcvPmTbKysqhatSrdu3dn1KhR+Pn5WR3/+++/M336dI4dO0ZOTg7h4eG8+eabtGzZ0k05UFEpHZS0PuNibRkX5DX7lzGPLPaOnEi0WOz1DuFMVLJdiz1HXrPu1po+fToLf1pCeJOOhD3dCJ3Og98vHOXsr1upFFaH8R8vw9PLYreZcP8m3348BK3Wg9bdRuDjV5ajeyK5H3OFBfO/p1WrVorElRebq7WcvWaiaoGYZUwprTy954edsEp7bUgofZ+vwOGTD7hwJR2j0Uyj8LJ0aBHE1RsPeeujaLJzbMW2L2mqaFxFJWXm25KPNbz7dZG/r6i4vWUsqr2hs1rdunUj9NnR+Pg9MkBp0fklygVXY/eGeRzbu5pWzw8DYOvKr8hIT+XNTyMJqWZxGYpo05ev/q8vkydPZuvWrULmsTj0VC33ae0/msSyDXd5mPFoZdzGnfHcvpfFsP6V6dGhPOu3xxV7XLIRZP6wVNweraj2hs5qNWjQwKoizqNRC4uF3v2YywBkZz7kwqnd1HzmufyKGMDbpwzPdRjE9evX+e2334TMY3HoqVru07p07aFVRZzHniOJAFQPK3jWkCvjkktJW4Hn9paxqPaGSlv/JSfeA0BvKAfA3VuXyM3JpurTjW2OrfpUI4D/VcZVXBaX0nkUNTZVSxmrygpBlumeSck5QsVVIGrLWB6i2hsqqWUyGdm5bi5anQeNW/UGIOWB5dHMPzDY5nhDkKW1cP++7eokUfMocmyqVtGtKrUaGNa/Mrm5ZnYdSpR8njstNEuVa9vjJCYmkpgo/SZIRVR7QyW1fl78BTevnOb5gROoULkGADlZlnnKOk/bdzx5eFpWPmVk2L4UVdQ8ihybqlV0q8qxI8KoV1vPwtV3iLkr/S0gbrX21GilbwLgsJvCbDYza9YsFi9ejNlsxmw2o9PpGD58OBMmSJ/D54jMLCOBPl529zljbyia1rZV/+Hw9qU06ziYjn1fz0/39LbMqDDm2D7y5eZYCruvr23fnIh5FD02VatoVpWjBoXQv1tFNu6MY/mGe7LOdWVchaHRieE5IRWHPwk//vgjJ0+eZNWqVRw9epRjx44RGRnJqVOn+PHHHxUJQFR7QyW0tq+Zxa71c2nabgADXvnYap8hwNIVkZxk2xWRkmjpwggOtu3CEC2PJSE2Vct5q8oRAyszfEBltu6J598/3JR9vlutPbUa6ZsAOKyM169fz4wZMwgLC8tPCwsLY9q0aaxbt06RAES1Nyyq1vY1s9i5djYRbfrzwmuf2ozYVgqrhYenFzevnLY59+bVM4DFT1rpuFylJXJsqpZzVpUjBlZm5AshbNsXz8z5N2Sf76q4pFKqLDRzc3MJCgqySQ8KCiI3t+C3IctBVHvDomjtWDubnWtn06R1XwaN+QytnZvt7VOGZxp34PcLx7hzIzo/PSsznWN7VlG9enUaNmwobB5dradqufdeDh9gqYi3709g+nc3nDb0cau1p0YjfRMAh33GnnYGl6Tsk4Oo9obOai1ZsoQda2YRUK4yT9dvyZnDG6326w3lqdWgFQDdhrzDlfNH+OFfY2jTfSTevnqO7okkJSmW6V/Oszv/UYQ8FoeequU+rb7PV2DUoBDux2dx8lwKnVpZN8iSknM4ea7gV7K5Ki7ZCNLilYrDyjg6OpqIiAibdLPZTHZ2tmJBiGpv6IxW3kKNBwl3iZz3oc3+GnWfy6+MywdXY+w/lrBl5Uz2bJyPMTeHkOrhvPLXebRt21bYPBaXnqrlHq06NS3+KcHlvflgbA2b/WfOp0qujJWMSy4lbQBPCG8KuThany+C1tqjxiJrDWhmKUhKXjNVS54WiFvGlMznH70pnKUgbwpnUMKbImPx55KP9R0+scjfV1TcvgJPRUVFxSUIMktCKqqfsYqKSqkkc9mXko/1GfqBCyORRrG2jAt6hNFoYHDfUPp1D6FSRR8eJGez+0Ac85dctzsh3NGjWmnTOnMllyoVtJTz15KYYmLKYttVeXlUrailR3MvqgZbBi6u3zOy6XAOdxJMzBhXRpG48mJT8l7ae0wOreRNl9ZBNG1ooHJFb7w8tdyNzWLfr0ms2RprV8vRY7II91JJrV6jz1mlhwZ7MbRPRZ6u5kOQvyc6nYa4xByO/5bK6l/iSUq2P/tp0/z6vPjedbv7Vs6objc9M8vEyIm2c45XzqiuaBkrMiWsZSxEN8Vbox/5nS5fe8vid9onlFo19QX6nT4pWk9X0XE7zoSvt+MvqxqsZVw/H5LTzfxy1DK42rqBJ+MH+PDNmoIrcCXzqKRe9/bl8z11dx5MzPfUfeXFUNo1DyzQU7c48imiVvlAT4L8PTh0MpWEpByMJjPVQ33o3i6Qds38eXPyFZJT5Y9lnL+ayc4j1oN1uSZ5hULpMiaZYljmPH78eHbs2MGiRYto3rw5AIcOHWLq1KncuHGDSpUq8Ze//IWePXsWquX2ylhUT1dRtP75w0MA3h/ii7eD2YQD2nhhNMG36zJJSbeU7tNXjXww1Je+rb1dnkel9UT11BVV60x0Omeibd/jeO5SOh+OrUqX1oGs3irv5bIAsYm57D8p/f2Qf8StfsYunk2xbt06MjOt34UZExPD2LFjmTRpEv369ePgwYO8/fbbhIaG0qhRI4d6bp+IJ6qnq6ha9ihn0FA1WMeZq7n5FTFASrqZM1dzqVXF/m0W2c9YVE9dUbUKIjbR4n2i93O+YtLpwNvLuUd+d/oZu9Io6N69e3z99dd8+umnVulr166ldu3aDB48GC8vLzp27EjHjh1Zvnx5oZpubxmL6ukqqpY9qla0FKYb92wrrxv3TDR/xv4/ksh+xgXhbk9dUbXy8PTQ4OujxdNTQ9XKPrwyyOJvcvw36fOCH6dFQz/aRlRDp9OQnGrk0Jl0lm9JIiNTWt+Ce/2Mpf+ApKSkkJKSYpNuMBgwGKyXcpvNZiZOnMjYsWMJCQmx2hcdHW1jY1C/fn02bdpUaAxur4wL8zttGO6Ph4dGkpnIk6BlD0MZS6FLTrc9316aq+JydT5d5albmspFt7aBjB32qIK4F5fNtO9vEXX5oWSNPC7fyOLI2XTuxefi662hyTN+9GhjILymD5O+uUtWtvvLvkNktHgXLlzIrFm2b4ieMGECb775plXa0qVLMZvNDBkyxOb4tLQ0nn76aas0g8FAWlrhHhwOK+Pvv/+eMWPGALBlyxZ69OiRv2/mzJm8++67hX5BYUj1O03LLXzw4UnQskeeI1au0bZA20tzVVyuzmeep+6CFbdd4qlbGsrF4dMpxNzLwsdby1NVfWneuCwGvXNtrr//567V530n0rl5N5uhPQPp2dbA2p3JhWq4ukw4RIbnxKhRIxkwYIBN+h9bxTdv3mTOnDmsWLHCro5eryc11fopJCUlBb1eX2gMDn86Nm/enP/3d999Z7Vv//79hYpLITPLiKeH/TCc8XQt7Vr2yGt1eOhsC5+9NFfF5cp8FtVTV8R76YrrlZCUy+kL6Rw5ncqSDbHM/CGGVwcFM7hHeVk6BbFhdzI5uWYiwqX12bu67DtEq5W8GQwGqlSpYrP9sTI+fvw4Dx48YODAgTRv3jx/BsW4ceP46KOPqFu3LufOWU87jIqKom7duoWH62jn4+tB/rg2RKm1IqJ6uujxvswAACAASURBVIqqZY+8QTv/Mrb69tJcFdeT4KkrqlZBXI/J4urNTHp1LFcknTyMJkhKzqVsGWkDgu71M9ZJ3yTSo0cPduzYwfr16/M3gM8++4x3332X/v37c/HiRVavXk1OTg579+5l9+7dvPTSS4WH62jn465hf3QQU+qNqqJ6uoqqZY+bsZaWRbVKtrezWiUtpgJ+OEX2M85DNE9dUbUc4eWlkVx5Foanh4agAA/Jc5bd6Wcsp2UsFV9fXypVqmS1gcVW2N/fn7CwMObMmcOPP/5I06ZNmTJlCp9//nmh09qgkMo4z7WtSZMmXLx4kYiIiPzPly5dkpwBR4jq6Sqqlj0SUszcjDXS6CkPDH6PfiQNfhoaPeXBlRj7j4Ei+xmDmJ66omoFGuz3CzesU4ZqoT5E/y5vAE/vZ79qGNI9AA+dhhPnpek9CX7GFy9ezO+uAGjVqhU///wzZ8+eZdu2bZIWfEAhA3gXLlwoUpBSENXTVRStLk0tKz30vhp02kefk1LNnLj0aInr+gPZjO3nw/gBPhz4zTLtq00DTzQa2HAom/eH2PbxiexnLKqnrqha44aHEOTvwZnodGITsvHy1PJ0NR/aNfMnI9PEgpXy+tlf6OJPrWreRF3JJP6BER9vDU3q+lK/li+XbmSxZX8J8DMW5EWjUnH71DYQ09NVFK0ezb3sfr5y22hVGV+/Z2L2ukx6NPeiR3MvzGa4cc/Iol9yuJtQ8ACJqH7GInvqiqi19+gDOrcMoFPLAPzL6jCbITYhh617k1j9SzxxidLnZQNEXc0ktJIX7Z/To/fTYTKZuRefy7LNSWzcm2J3qpqr8ygbQd7gIRUhKmOTCZavi2H5uhhV6w+8N1v6UtQb903M3ZBZ+IEKxOVqvWnzbjBtnnN9xPYQ4V66UuvA8RQOHLddtOAsx6MyOB5VsKeJHJQuY1Ixq+byKioqKu4nY/cSycf6dhzmwkikIUTLWMX1XLt2jQ0bNnDw4EFu3rxJVlYWVatWpXv37owaNQo/P78Cz126dCmTJ08G4PDhw3ZfUquiIhxqn3HBFOQ1+5cxjyz2jpxItFjs9Q7hTFSyXYs9R/6wqpZ9S8L+7Y+x6KclNGnWjh4vdEWn8yD63HG+/vprVq3bxKSpP+Ll7WNzXlJiHP+aNh1vHz+yMh9y9FI63VsEKXovHfnpFmThePi07Wh+QX66zsYF4t1LpctFQd7UImgVFbPaZywPUS0JS5tWt27daNR+GH5lHhmzdOw+iODKVdm4agH7d66nc0/btfaLv5tKhUpVCA17isN7N9vsd1U+8yiqhaPScala7tOSTQlrGbs9WlEtCUubVoMGDawq4jyatXkegNs3rtrsO3FkF6eP7WPkGxPRSpgY7yq7xKJYOCodl6olnk1ogRTTPGOlcHvLWFRLwidBCyAx3tIqMQRY9wNnPExjyff/okPXgdSsXZ/dWyOLPTYouoWj0nGpWu63CZVKSZtN4faWcWEWe4H+XnjYWdeuahVdy2Q08nPkfHQ6Hc3bdbfaF7noP5jNJl4YPkGSltKxgcXCMXLbA2YuimPW0jjOXcmkRxsDn4yvLKulLOr1V7XklwlZuNBc3hU4bBnfuXPHxjxZaUS1JHwStJb9MIOrF8/ywvDxVA6tnp9++cJp9m5bw5h3PrPbtVEcsYEyFo5Kx6Vqud8mVCpmQSpZqTiMdvz48fl//9FgWSlEtSQs7Vprls5m5+YVtO86kF4vvJqfnpuTw8I5Uwhv2IwWbbs7UHBdbI6Qa+GodFyqlvttQiVTwvqMJVto3rp1yyUBiGpJWJq11i2fx8bIBbTp1JeRf55otW/XlpXcvX2drn2Hc//urfwtM8MylSz+/p0Cy0Jx2CXKtXBUOi5VSzyb0IIwa7SSNxFw2kJTKUS1JCytWuuWz2PDiu9o3bE3fxr/D5v7Gh93F7PJxFefvsmH4/rnbyeO7ALg07+NpG/fvi7PZ0HItXBUOi5VS0ybULu4wM/YlThtoRkREaFIAKJaEpZGrQ0rvmPDiu9o2aEXr0z4yO50tTad+jLur1/abHXrNwXglQkfMW3aNJfnUykLR6XjUrXEso91hFmjkbyJgGqh+YRoLVmyhHXL51GuQiXCGzbj131brfYbAoKo17gFVWvUpmqN2jbnnzluec1W4+fa0qVFGB//23allZL5VMrCUem4VC33aclGkO4Hqbh9njGIaUlY2rR+++03ABLi7rHgPx/Z7K9Tryn1GreQF4hCsdlDSQtHJeNStdyrJQczYrR4pVKsrm0F+QbIxdH6fFWrYL2D55Xpn2sdrlc0roK8KeTiyJtCLqLeS1eUC1G1isqDU7skHxvQpFORv6+oCNEyVlFRUVGcEtZNofoZq6iolEoSz+6XfGxQw7YujEQaQlhoajQwuG8o/bqHUKmiDw+Ss9l9II75S67bnRDu6FFN1bI/gb6gR0hntcb8bR73b0Vx70YUyQkxGIJCGTvF9rEwOSGGuZM629XJo2yl1/AxNLdJd+aaiVzGnh92wio9tJI3XVoH0bShgcoVvfHy1HI3Not9vyaxZmusXa3tS5oqFldebKJqFRlBZklIRYhuirdGP/I7Xb72lsXvtE8otWrqC/Q7VbXcq7Vv/Ux8ygQQHBZOVkbBMxt89UH0/tO/7O7bvuJTTLmZeJUJFzafrtTq3r48fZ+vwOGTD9h5MBGj0Uyj8LK88mIo7ZoH8tZH0WTnSA9MxDwqrSUHURZzSMXtlbGo3qmqlmOtNz7ZQUCFMAAWfNKb7Cz78369vP2o17yfTfrt30+RlZFKt27dOHnN1v9ClHy6Umv/0SSWbbjLw4xHrcONO+O5fS+LYf0r06NDedZvjyvReXSnn3FJm03h9p8OUb1TVS3HWnkVsbOcOWix5Bw8eLDisZUUrUvXHlpVxHnsOZIIQPUw6f4boubRnX7GpWo5dHEgqneqquU6r9nszHQuntiCISiU1q1buzw2UbUKokKQFwBJyTluiUtULdmUJqOgHTt2sGTJozesDh48mM6dO9O5c2e2bt3q4EzpiOqdqmq5zmv2wonNZGc9pEHLgQW+QUTUfLr6mmk1MKx/ZXJzzew6lCj5PFHz6E4/Y5NGJ3kTAYeV8fz58+nU6dFk6OzsbFatWsVPP/3EsmXLFAlAqt+pqiW2lhzOHlyFRqOlYauBxRKbqFr2GDsijHq19SxcfYeYu1mSzxM1j+4qY1DKuilycnKoXLly/uemTZsSGBhISEgIGRkZigQgqneqquUar9n4u1e4c+001Z9phSGo4BcXiJpPV16zUYNC6N+tIht3xrF8wz1Z54qaR3f6GZvRSN5EwGFlnJJi3c/zz3/+M//vxETpj1COENU7VdVyjdfs2YOrAGjY2v7AnStiE1XrcUYMrMzwAZXZuieef/9wU/b5ouaxNPoZz549my5dutC0aVOaN2/Oa6+9ZmWqdv78eV566SUaNWpEhw4dWLRokSRdh1E0bNiQlStX2qQvX76chg0byspAQYjqnapqKe81a8zNJurX9fjpg6jVyPFCEFHz6YprNmJgZUa+EMK2ffHMnH9D9vlKxyWqllxcZaHZo0cPVq9ezYkTJ9i/fz+tW7dmzJgxmEwm0tLSGD16NG3atOHo0aN8/fXXzJo1S9IYm8PKeOLEiaxZs4YRI0YwdepUpk6dyogRI1i7di0TJ050dKpkRPVOVbWU95q9fHYXD9MSqde8HzqdZ7HFJqoWwPABlop4+/4Epn93w+kFEKLm0Z1+xnIG8FJSUoiJibHZ/tg7AFCjRg38/f3zP2u1WuLi4khNTWXbtm1otVrGjRuHt7c3jRs3ZvDgwSxdurTQeB0u+ihXrhzLly/n8OHDXLlyBYD27dvTsmVLudelQET1TlW1HGud+3UdKQl3AHiYlogxN4dDm2cDYCgXQv3m/W3OkdpFIVI+XanV9/kKjBoUwv34LE6eS6FTqyCr/UnJOZw8J823WdQ8utPPWE5f8MKFC5k1a5ZN+oQJE+y+/3PPnj28//77pKamotFoeOWVV/D39yc6Oprw8HCrWUL169cnMjKy0BgkrcBr2bKlohXwHxHVO1XVKvi8swdXc+vyUau0/T//G4CwWs1sKuOUxLtcv3CQ0JpNKF/5qRKTT1dq1anpB0BweW8+GFvDZv+Z86mSK2Ml4xJZSw5y+oJHjRrFgAEDbNINBoOdo6FDhw4cP36cBw8esG7duvyJDmlpaZQtaz132mAwkJZWeHeM25dDA5hMsHxdDMvXxahaJUTr5Xd/knW8Iagyf5st780xIuTTlVrT5t1g2jzn+ojtIWIeldaSg5yWscFgKLDidURAQAAjR47kueeeo2bNmuj1ehISEqyOSUlJQa/XF6qlWmiqqKiUSq5dvSL52BpPPe309+Tm5tK0aVNmzJhBSkoKX331FXv37s3vqpg+fTpnz54tdFaFGLOdVVRUVBTGVfOMFy1aRFycxcApMTGRyZMn4+XlRePGjenatStGo5E5c+aQnZ3N2bNniYyMZOjQoYXqFmvLuMOgw3bT96xqSXJqDpd/T6d2TT0PM3J5adypAnX2rLL0X9vzTv3LmEd2fUdOJFrs+nqHcCYq2a5dnyPfWme17OXT2TwqoZV3jpJaBfkGO3vNEvYf49Z862mUppwc7kZuyf9coWtbntswj/SrN4n5cTXZCUnow2tRdfSLmLKy8QoKUDwuJe+laOU1T++PPsuvDQnNt/a8cCU939qzQ4sgrt54WKC15/YlTRWNq6hcuXpN8rFPP2XbZ18Q48aN48yZM6Snp6PX62nQoAETJkygXr16gGWe8eTJk7lw4QKBgYG89tprjBw5slBdIfqMh447yd1Yy9LP/85shK+Pcw12ka3/lMqjyFpFuWYPr93i9tINjvX/Mgqz0cihdkPJSUjKT087f5mG86a4JC57lPbyKqq1p1zMLnrwnz17tsP94eHhrFixQrauEN0UeQW7qIhs/adUHkXWKuo103h6oivjV+B+D4MeY2Y2OUnJVumZdxz/M4t6L0Utr6Jae8qlVC2HLmmUGuu/EkpRrlnlgd3onnKa7g9O0eX2Iep9PQkPg/UIdNy2A3ga9DT+75eUbVgH75CKlH++DeH/+j9Szxc8WCPqvSxp5dXd1p5yKWmVsRDdFEpRmF1fw3B/PDw0ktbCK6n1pODsNUs6eoZ7q7aSfvUGHgY9Fbu3p/r4EQS1a8ahti9hTLe8ReTql/PwrliOKn96gdCX++afH7t5D6eGv0u3xJOKxuVqSlJ5dZW1pyuvvSiVrFRKVWUs1a4vLddYrFpPCs5es0OtX7T6fHvxelJ+u0jdz96lxpsjuTJ1LgBmo4nM2/eJ33mI++t3kJ34gKBWEVQbP4ImS75SPC5XU5LKa56154IVt11i7emKa1+qKuNPP/0UjQMTjUmTJikeUFHIzDIS6ONld58z1n9KaT0pKHnNfp+xgFr/mEDFnu3zK+NGP0wlsEUT9jbqhSnTUiHcX7+D9Ks3afDt5GKJS0lKSnktqrWnu669yVyyemEdRlu/fn3q1atHvXr12LVrV/7feZtolBbrv5KKktfMnJtL1p1YPMsFAuATVpnQl/sSu2VPfkWcx91Vjh2xRL2XJaG8imTtKZeS1mfssDIeMGBA/mYwGKw+21vH7W5Ki/VfSUXJa6b19sKnSjDZsZalpT6hwQBodLavyNF4OH5tjqj3UvTyKpq1p1xKVWX8OI66K0ShtFj/lVSUvGa1J7+N1tOT+xt3A5B+8Rqm3FyC+3bBw996BL7KyIJf36R0XEoicnkV0dpTLmazRvImAkIM4D3frjyVKngD4G/wwNNDy4gXLDfvXlwW2/fFS9IR2fpPqTyKrOXsNWu1fzkJe34l49ZdPMr4UaFHe8p3bEHSr6e5/q3FkCgnKZnr/1lIzXdfo+2xddxcsJKcpGQCW0UQOrQP6VduUObpaorGVRClvbyKau0pF5MgLV6pOKyMmzRpkt8izszMJCIiAgCz2YxGo+HkSftTieTSq3NFGtfzt0p7bWhVAE5HJcuqEES1/lMyj6JqgXPXLDcljSojBuBZLgCz0Uj6lRtET5rJta//iykrO/+4Cx/8i7RL1wh7dTBP/98baL29yLx9nxvzlnH5k294/u4RReMqiNJeXkW29pRDSRvAc1gZnzrl2KNAKd7+6HzhB0lEVOs/JfMoqhY4d82O9hot+dhbCyK5taBwo24l4iqI0l5eRbb2lIMofcFSEaKbQkVFRUVpROkLlorqZ6yiolIqOXbxgeRjn6sT4MJIpFGsLeOC7A01GhjcN5R+3UOoVNGHB8nZ7D4Qx/wl1+1OCHdkI6hq2Z9Af+Dn9rz43nW7+1bOqG43PTPLxMiJtnNLV86orui97D/ukl2tx/Hy1PCff1SjUnkvNu15wPcrbQd+1s2uzZ8+th2d79+hDP07FPymhVyjmdGfWuv9+LFlKp1o99IV5cKeTejLA0KoXVNP7ZplCAn24V5sptM2rc5qFZWS1jIWopvirdGP/E6Xr71l8TvtE0qtmvoC/U5VLWW08jh/NZOdR6wHZXJN8oVcERvAy33K4a93rrgev5DF/UTb5bZhwR70bF2G05fkubCJei+V1Hp9WLV8z2a9X9GqCSW15FDS1se6vTIW1dP1SdB6nNjEXPafTJd9XnHEVjPMmz4dA1m4No5XB8m3XIy5n0vM/Vyb9Nq9LfOV95/MkKwl6r0U1bNZaS05lLTZFG6PVlRP1ydB64/odODt5fyjnSti02pg/LBgTp5P58hp5VZreXlC8/o+JCQbOXslu/AT/oeo91JUz2alteSgLvqQiaierk+C1uO0aOhH24hq6HQaklONHDqTzvItSWRkSn+2dUVsfToHEhrsxZff3ZF9riOa1fPBz0fLjl/TZD2+i3ovRfVsdiclbWqb21vGhfmdBvp74WHHZETVKrpWHpdvZBG57QEzF8Uxa2kc565k0qONgU/GV5bVUlY6torlPBjaqxwrNicQm2jbzVAU2jbxxWQ2s++U9C4KEPdeuqJclHRMZumbCLi9ZSyqp+uToJXH3/9z1+rzvhPp3LybzdCegfRsa2DtzuQCznRtbGOHBnM/PocNO5MKP1gGlcrpqFPNi6jfs4h/IG+YR9R7KapnszspaS1jycuhH0fJ5dCiero+CVqO2LA7mUFdA4gI95VcGSsZW/tmZWlU14+/f3ULo8LD4u0iLO9w2ydj4C4PUe+lqJ7N7kSUvmCpuH05dHxiNtXDyuDpobF5xHLG01XVUsYf1miCpORcypZxbE/pitg8PDS88kIFTkSlk5RipFIFTwDKBViKaxlfLZUqeJKaZiTdzoszHaHVQutGPqQ+NHHigvyBJVHvZXGVi5KEsYRVxm7vMxbV0/VJ0HKEp4eGoAAPklOlP9YqFZu3p4aAsh4810DP3Mk18rcp74QB0KG5gbmTa9CltX8hSrY0ru2Nv17H4bMZOPPELuq9FNWz2Z2UtNkUbq+MRfV0fRK0APR+9ovAkO4BeOg0nDj/sNhjy8wy8eX3d2y2ucss55+ISufL7+9w7Kz8CuZRF0Wm7HNB3HspqmezOzGbpW8i4PYBPFE9XZ8ELYAXuvhTq5o3UVcyiX9gxMdbQ5O6vtSv5culG1ls2S/dKlGp2IwmOHzKtqKtGGQprvficuzuL4yAsloaPO3F1ZgcYmKdm50h6r0U1bNZaS05lKoBvOJCRE/XJ0Ur6momoZW8aP+cHr2fDpPJzL34XJZtTmLj3hS7U6WKKzaladPYB51W49TA3eOIei9F9WxW2jNbKqJMWZOKEJWxiJ6uT4rW8agMjkcVrXJ6HFd618Ym5koyFSqIjfsfsnG/9G6XghD1Xorq2ay0Z7ZUTKaS1TJWLTRVVFRKJRtPSu+K6h0hvV06bdo09uzZw927d/Hz86Ndu3b89a9/JTAwMP+Y8+fP88knn3DhwgUCAwN59dVXGTlypENdtw/gqaiIyrx583jrrbfo3LkzderUoVOnToWes27dOl566SUiIiJo0qQJvXv35ttvvy2GaFX+iKsG8HQ6HdOmTePXX39l/fr13Lt3jw8//DB/f1paGqNHj6ZNmzYcPXqUr7/+mlmzZrF161aHukL4Gf9lzCPrvyMnEi3Wf71DOBOVbNf6z5Gnq6pl3yrxwM/tXa7lbD5F1Zo5cyY+fgFUrBKOt18qKRkwx8H/07alH3Lh6DqebtSVZj37otFoSUmI4c4di6+GPT/plTOqF2hfevi0bZdKnvd0SSoXzmoVFVdNWXv33Xfz/y5XrhwjRozgvffey0/btm0bWq2WcePGodVqady4MYMHD2bp0qV07969QF239xmLaiOoasm3XRQ1Nme1duzYwcYoy9zmn77oTXZ2wf3N5w5Hcv7XNXQb/iXPPNffat/Ygv//APHsS0XVkoucAbyUlBRSUlJs0g0GAwaDwc4Zjzh8+DB169bN/xwdHU14eDha7aOOh/r16xMZ6fjdjW7vphDVRlDVkm+7KGpszmqFhYVJ0jebzRzb8R0Vq9TLr4izM9OQMxwjkn2pqFpykdNNsXDhQjp37myzLVy40OF3bN68mcjISP7+97/np6WlpVG2rLVLnsFgIC3N8XRMt7eMRbURVLXk2y6KGpur7SWTYn8nOf4mjdoO59dfvuXUnkVkPnyAl4+eOhG9GNn2A8qUKVPg+aLZl4qqJRc5y6FHjRrFgAEDbNIdtYo3bdrExx9/zJw5c6hXr15+ul6vJyEhwerYlJQU9PqCX/0FDirjgkyCALy8vKhatSrvvPMOLVsW7V1VhVn/NQz3x8NDI2ldvarlPi2RY1M6n38kKfYaAJdObcZozKF517EYgqpwLWoPvx1awZ//fI1FixbZPffyjSyOnE3nXnwuvt4amjzjR482BsJr+jDpm7tkZavX3lnkDMxJ6Y54nMjISKZNm8bcuXNp2rSp1b66deuyZcsWTCZTfldFVFSUVVeGPQqsjB2ZBBmNRi5fvsz777/Pxo0bJWfAHqLaCKpa8m0XRY3N1faS2ZmW/t6MtEQGjvsvVeu0AqBW426YMXP06Fr27dsHVLM5V0T7UlG15OKqSbuLFi3i22+/ZcGCBTRo0MBmf9euXZk+fTpz5sxhzJgxREdHExkZyccff+xQ16k+Y51OR926dRk+fLgzp1uRmWXE08N+GM7YCKpa7tESOTal8/lHPDx9AND7B+dXxHmEN7P0IR89elSy3obdyeTkmokI95V8jqjXy9XX3hEms0byJocpU6aQlpbGyJEjadKkSf6WN2tGr9czf/589u3bx7PPPsubb77J+PHj6dGjh0PdIvUZv/TSS0U5HRDXRlDVkm+7KGpsrraX1AdUAsDPUN5mXxlDBQC7I/UF4U77UpG15OKqlvHFixcLPSY8PJwVK1bI0nX7bApRbQRVLfm2i6LG5mp7yfIhtdF5epOWbDtFK+2BxS0tKChIsp477UtF1pKL0SR9EwG3V8ai2giqWvJtF0WNzdX2kp5evtRq2JWHKXFcObPdat/ZA8sAaN/edhGDiPalImvJpaT5Gbt9apuoNoKqlnzbRVFjc1Zr3bp1/LrX0g+YkZaI0ZjDr7/MBsAQFGK1uKNV73e5eekwW356j8bXh2MICuX6+X1cO7+H/v37ExERAUuuW+mLaF8qspZcSprrjtsrYxDXRlDVkqclcmzOaK1evdpm4O3w5n8DEPp0M6vK2BAUwpB3VnBo01ec/3UNWZlp+JcPo22/D/jiiz/Z1RfZvlRULTmoFppOIKqNoKrlXj13a/30008OvSj+iH+5KvQYOcMmXVtAZ6DI9qWiaslBbRmrqKioCEBJq4xVP2MVFZVSyfc7pB87povr4pCKEBaaGg0M7htKv+4hVKrow4PkbHYfiGP+kut2J4Q7spcUQavDoMM2+14eEELtmnpq1yxDSLAP92IzeWlcwasc96xqqZhWnl6v0ees0kKDvRjapyJPV/MhyN8TnU5DXGIOx39LZfUv8SQl2zfn3jS/vqJxPT/shE16aCVvurQOomlDA5UreuPlqeVubBb7fk1izdZYu9d/+5KmipcxJe9lQZgyM4h5e5hVmkdwCAEDRuBTqx54eJB983eSNy4n+J1PAGXKa14+lbyXSmoVFZMgU9akIkQ3xVujH/mdLl97y+J32ieUWjX1BfqdljSt14dVIzk1h8u/p6P3K9plV0qrfKAnQf4eHDqZSkJSDkaTmeqhPnRvF0i7Zv68OfmKrLmuSuaxe/vy9H2+AodPPmDnwUSMRjONwsvyyouhtGseyFsfRZOdI/0GiHIvMy9HkbbfevobRutr7FE+mOC/fgEmIynb1mHKfIi+dRcqvvVPh9qi5NGVWnIoac/8bq+MRfVOVdqHdei4k9yNzQLgvzMb4evj/BRvpbTORKdzJtrWR/fcpXQ+HFuVLq0DWb1V+ssilczj/qNJLNtwl4cZj5o3G3fGc/teFsP6V6ZHh/Ks3x4nSUuke5kbf5+HR/c5PMZ/wHC0fn7c+/yv5MRcByD9yB4q//PfeJSrYPcckfLoSi05lLTK2O2LPkT1TlXahzWvMCqBklr2iE3MAUDvJ305Ligb16VrD60q4jz2HEkEoHqYdN8G4e6lzgONt4/dXRovb/waPkfWpaj8ihjAnJVJ2sGCO0GFy6OLtORgMkvfRMDtLWNRvVPd6cNa3Hh6aPD10eLpqaFqZR9eGRQMwPHfpC86KC4qBHkBkJScI/kcke6lX5OWlGnWHo1OhzElmYcnDvJg/VLMmZbVdp6h1dB4epH1u63/Qfa1gt+MLVIeRUHe3AT3r8JzWBnfvXuXypUr2923e/duOnbsWOQARPVOdacPa3HTrW0gY4eF5H++F5fNtO9vEXW56K+1VxKtBob1r0xurpldhxIlnyfKvcy6domHJw+TG3sXrY8fPvUjKNuxJ961wrk/bSLmrEx0An0BIwAAFc9JREFUARYPC+MD2/zlPkiwSctDlDyKhFF5V06X4rAyfuWVV5g/fz5VqlSxSl+1ahVz585VpDIW1TvVnT6sxc3h0ynE3MvCx1vLU1V9ad64LAa92x+abBg7Iox6tfUsWHGbmLvSH31FuZf3v/w/q8/pv+4h5/YNAvoPo2ynXqRsWY3GyxsAc65ty9+cU/DTgCh5FIlS1Wf84Ycf8tprr3H9+vX8tHnz5rFw4UIWL16sSACieqe604e1uElIyuX0hXSOnE5lyYZYZv4Qw6uDghncw9YS0l2MGhRC/24V2bgzjuUb7sk6V+R7mbJtHeacHHzrW94WYc62/MhoPDxtjtV42qblIXIe3UVJ6zN2WBm3b9+ejz/+mDFjxnDp0iWmTJnC7t27Wbx4MZUqVVIkgPjEbPwNnnh62PbZOOOdKqJWSeN6TBZXb2bSq2M5d4cCwIiBlRk+oDJb98Tz7x9uyj5f6HtpMmJMTkSrt1hM5nVP5HVXPI5HQMH3Q+g8ugk5LyQVgUJnU7Rs2ZIvvviCESNGcOvWLRYuXIi/v79iAYjqnepOH1YR8PLSyDI3dxUjBlZm5AshbNsXz8z5N5zSEPpeeniiCyyHMeUBADl3bmLOyca7Zh2bQ71q1C5QRug8ugmzySx5EwGHlXGTJk2IiIhgzJgxpKenc+TIEVq2bJmfrgSieqe604e1uAg02O8XblinDNVCfYj+3b0DeMMHWCri7fsTmP7dDadbMCLfy4C+Q9HoPMj47ThgmcKWcfY43rXr4RlaPf84jbcP+tYFr9kVOY/uoqSZyzscpXH0UlKlENU7VWkf1ufbladSBcvgjL/BA08PLSNesPzj3IvLYvs+6YsrlNIaNzyEIH8PzkSnE5uQjZenlqer+dCumT8ZmSYWrJTXN6tkHvs+X4FRg0K4H5/FyXMpdGpl/dielJzDyXPSpt6Jci+D//YFmRfPYUyMQ+Pji2+9CHzqNiDr90uk7d6cf9yDdYvxrtuAin/5J6k7f8aUmYG+dRe7XRei5dHVWnIwCdLilYoQQ+aieqcqqdWrc0Ua17Pu3nltaFUATkclyyqQSmntPfqAzi0D6NQyAP+yOsxmiE3IYeveJFb/Ek9covS5vErGBVCnph8AweW9+WBsDZv9Z86nSq6MQYx7acp4SJmWHdCVKYvZZCI39i4P1i0hZccGeGz2RG7cPe5Pm0jAgBEYug3M96aI/eZTgt/+WOg8ulpLDqL0BUtFiMpYVO9UJbXe/uh84QcVs9aB4ykcOC79RZmFoWQep827wbR5zvUR20OEexn3zaeSj829d5v4OVNl6YuQR1dryaGkVcaqhaaKikqp5NNl9l0H7fGPoe5vl7o/AhUVFVnMmzePqKgooqKiiImJITQ0lF27dtkcFxMTQ+fOnR1qTZs2jb59+7oqVLdiFmRgTipC+Bn/Zcwj678jJxIt1n+9QzgTlWzX+s+RB7GqZd8q8cDP7V2u5Ww+RdUC8e7lgZ/bM3PmTHzKBBAcFo6PXyqpGfCDbV1MdlYQvf/0L7vXY/uKT8nNyaRNmzbClrGiYjSWrId+t7eMRbW9VLXk2y6KGltp09qxYwfbL4cBsOCT3mRn2Z+C6OXtR73m/WzSb/9+iqyMVOpEdCMoyHaGhgh5VIKS1gOrWmiqWorZLooaW2nTCgsLk/wd9jhzMBKAhq0HKxqXq7XkUtKWQ7u9ZSyq7aWqJd92UdTYngQtqWRnpnPxxBYMQaHUqNva5XG509pTlJV1UnF7y7gw679Afy887Ky3V7XE0hI5tidBSyoXTmwmO+shDVoORKO1/+9f0vOYR6nzpiiIH3/8UZEApFr/qVpia4kc25OgJZWzB1eh0Whp2GpgscTljjzmYTKZJW9y2LRpEy+//DIRERHUqWPrIXL+/HleeuklGjVqRIcOHVi0aJEkXbdXxqLaXqpa8m0XRY3tSdCSQvzdK9y5dprqz7TCEBRS4HElOY+PYzKaJW9yMBgMvPzyy0ycONFmX1paGqNHj6ZNmzYcPXqUr7/+mlmzZrF169ZCdZ2ujJUaqRTV9lLVkm+7KGpsT4KWFM4eXAUUPHDnirjcae1pMpslbykpKcTExNhsKSm2K1Tbtm1L79697Q6kbtu2Da1Wy7hx4/D29qZx48YMHjyYpUuXFhqv05WxRqNMP4+otpeqlnzbRVFjexK0CsOYm03Ur+vx0wdRq5HjhSAlNY9/xGw2S94WLlxI586dbbaFCxfK+s7o6GjCw8PRPtYfX79+faKjows9V5KF5h+3Jk2aEBurzNxAUW0vVS35touixvYkaBXG5bO7eJiWSL3m/dDpCn5jiNJxudPaU06f8ahRo9i5c6fNNmrUKFnfmZaWRtmy1jNEDAYDaWmF/+ioFpqqlmK2i6LGVtq01q1bx6EDdwB4mJaIMTeHQ5tnA2AoF0L95v1tzpHaRSFKHpVATk+qwWDAYDAUfmAh6PV6EhKsXxybkpKCXq8v9Fy3zzMGcW0vVS15WiLHVpq0Vq9ezdGjR63S9v/8bwDCajWzqYxTEu9y/cJBQms2oXzlp0pEHpXA6AbX+Lp167JlyxZMJlN+V0VUVBR169Yt9FwhKmNRbS9VLffqqVr2+emnn+x6URSEIagyf5t9weVxFYeWHFy16MNoNJKbm0vO/97WnZVleYmsp6cnXbt2Zfr06cyZM4cxY8YQHR1NZGQkH3/8caG6bl/0oaKiouIKXPUOvPXr19OwYUNee+01ABo2bEjDhg05duwYer2e+fPns2/fPp599lnefPNNxo8fT48ePQrVVf2MVVRUSiVjpz2QfOycvwa4MBJpCGGhqdHA4L6h9OseQqWKPjxIzmb3gTjmL7lud0K4I0tCEbRefO+6zb6VM6rbzXtmlomRE21fP593fIdBh232vTwghNo19dSuWYaQYB/uxWby0jjHg617VrVUJI/g2KrSmWvm6jw6GxcoWy76j7vkMH6wTPn6zz+qUam8F5v2POD7ldYDXOtm11YsLoCf/lOV735cztlTR7l/9w45OVkEVwqlRZtO9Oj3Ij4+vjbnnDx2iM3rVnDtykVycrMJKleRhk2e45uZXyhaxopKSfOmEKLP+K3Rj/xOl6+9ZfE77RNKrZr6Av1OS5oWwPmrmew8Yv3etlwnCszrw6qRnJrD5d/T0fsV7RYqnUel9JTMo5JxKa31R17uUw5/vXP5dSau1atXs3n9Cpo2a0ObDl3R6TyI+u0kK376jsP7d/HZjO/w8vbOP37V0h9YtXQBjSKaM2jYa3h7exMfd5+b168qGpcSlLSHfrdXxqJ6p7rChzU2MZf9J9NlnWOPoeNOcjfWMmjw35mN8PVxrutfZD9jpfKodFyu9OetGeZNn46BLFwbx6uD5FlLOhtXt27daNllEH5lHk29er7nAFaEhLF2xUJ2bfuZ7n0GAfDb6WOsWrqAwcNG88LQV1walxK4YzZFUXD7AJ6o3qmu8mHV6cDbq2irF/MqqaIisp+xUnlUOi5XlQutBsYPC+bk+XSOnJa/Ks3ZuBo0aGBVEefRsq1llV7Mjd/z09atXIR/QCD9XxwBQGbGQ0wmxxWeO/2MXTWA5yrc3jIW1TvVFT6sLRr60TaiGjqdhuRUI4fOpLN8SxIZme4pDCL7GSuJ6OUCoE/nQEKDvfjyuztOna90XAnxltaqf6DlTSCZmRlcOHeGJs+2YPe2jaxe9l+SEuPx8vamabM2jHr9baC8y+OSgyiVrFTcXhkX5nfaMNwfDw+NJDMRUbUALt/I4sjZdO7F5+LrraHJM370aGMgvKYPk765S1Z28RccpfOotJ5SiFwuACqW82Bor3Ks2JxAbGIuFYPk/1sqGZfJaGTN8h/R6XS0bv88APfvxGAyGbl8MYqzp47Rd9BwqtV4muioM2z5OZKb16/S4ud1Lo1LLqbS1Gf85z//2eHJc+fOLXIAUv1O03KNJVYL4O//uWv1ed+JdG7ezWZoz0B6tjWwdmeyJB0lUTqPSusphcjlAmDs0GDux+ewYWeS5HNcGdfC7//N5ehzvDTyDUKqVAMgI8Pynr2U5Ae8/uYHdOpmeaN0s1bt8fUrw+plP7B27VrA2oPCnWWiVLWMT58+TeXKlenVqxeNGjVyyehkZpaRQB8vu/uc8U4VUasgNuxOZlDXACLCfd1SGSudx+K4Zs4gcrlo36wsjer68fevblGU8Sal4lrx03f8snE1nbv3o/+LIx9p/G9GhUarpW2n7lbntO/cg9XLfvjfEu0BLonLGUrabAqHA3gHDx7knXfe4fLly0yZMoWDBw8SGBhIs2bNaNasmSIBiOqdWhw+rEYTJCXnUraMrkg6ziKyn7GSiFouPDw0vPJCBU5EpZOUYqRSBU8qVfCkQjmLq1oZXy2VKnhSxrfwcXYl4opcsoC1KxbSoUsvRo//q9W+oHKWgTa9viyentaVa0BQOQC73r/uLBPGXJPkTQQc3mWdTke7du348ssvWblyJdWqVWPEiBEsXrxYsQBE9U4tDh9WTw8NQQEeJKcW72N7HiL7GSuJqOXC21NDQFkPnmugZ+7kGvnblHcspuUdmhuYO7kGXVr7uzyuyCULWL3sB9p17sHrb/2fjV95QGAQ5SsEk5aaQlZmptW+xPg4AIKCghSPqyjI8TMWgUJ/crOzs9m2bRvvv/8+S5YsYcSIETz//POKBSCqd6qSWno/+5d5SPcAPHQaTpx/KFlLSUT2M1YSUctFZpaJL7+/Y7PNXWbROBGVzpff3+HY2cIrrKLEtXrZD6xe9gNtO3bnz3+ZaGWM/jhtO3XHbDazY6v1QN32zWsBaN/edtWcO8uE2WSSvImAwz7jv/3tb1y+fJl27doxYcIEateurXgAonqn/n975xrT1BnG8X/HRdpyc4A4m6ljYsALAcoMmk0XYM5VR0QWYC7CopgtQeMn4odlF40zEhcvE2Uql23IJAI6rGMZMBVFlGUgKF2cwnDopotMZVXAS0/3oaMTe6GnfbFv7fNLzpfzln+f57R5OH3P8/5fllqpSQEInzQGms5B9N7WwWeMBDERUswIl+Li7/fw/UntyCKP8NrcYIwPMczhBfh7wsvzGSxLNXzZr9+4h7oTvU88R9Z6rHJkHRdLLZ0AnD5rWmiHuimu33hgdpxlXGVlZagoK0JwSChmRsfhVEPdsPGAwLGIijFMSb6Z+g6aTx3HvuKduPbHFUx6YQp+/eUcGo/XYnqUEiqVCnl7GpnExQKxG406G6vF+PDhw5BKpbh8+TJKS0uN5/V6PSQSCVpbW5kEwat3KistTdcgFOO9Me8lX/jKPCAIelzvfYj9NbdwpOEfs20/1liYOA7R04f/dF3x9kQAQJumT1Sh4tXPmGWOLONircUSe+I6f/48AKD3xl/YtXWDyXjkjBhjMZbJ5PgkbxcO7NuLluaTOFZ3BEFB47A4LRNLMt6Fh4f5Zx/Oul68TD/YyhN1bbNk4iIWayYuPGiZMwoSizWjIHuwZqIjFmtGQfZo8ZojwPZ7YYtR0EhYMwqyh0b1PJy9JO4fmyViwoOZX39HeGvNbyO/6D8qt4c5/H6O4vRFHwRBEKOBoOdjLthWyM+YIAhm7N69GxqNBhqNBlevXoVCocDRo5a3JWloaEBxcTE6Oztx9+5dhIaGIiEhAStWrEBwsOnyajGkrLpk82sP5Yc79F4s4MLPeM3K/y32zrTcNFjsLZqAdk2fWYs9az8hScu8JaGlqQWWWvbmyasWwPazNDcdc7xyttEqdGqYL/oHHlr1bT5eOZtZXEOxsdTasmULpPJAjJ84DT4yLe4MAqUnzOfSeuIAako/xHOTpiMmYSW8xkjx5+XzKPnyK9TW1kKtVkMmk1m8FiPxVK3AexLwam9IWs610HQHLYBPO1RHtHI21mNsiKFPevfHi3D/nuW2zTO1RfANCEHW2v3w9DJ0zsTOTYfcLwinar5AU1MTkpKSbIrZHK72o58sNEmLSwtNd9AC+LRDdURrqBDbwv3Bu/CRBRgL8RB+gQZ9qdR0lxExCIJg88EDTi/GvNobkpZzLTTdQYslrphj2LSX0XutE3UHNqH3Whf6bl7DhdZaNH5XgFmzZiE+Pt4hfUGns/ngAacX45Es9sYGeMPTzLp20uJLi+fYeNViiSvmOD/jA0QqF+CnH7/GFx+psGPtq6gsWI0XZ7yC4uJii33LtkLm8iLh1d6QtJxroekOWixxxRw9PDwREDQBETFJCI9KgNcYH3RpGtHeWIV16zywYYPpIhQx8FJkbcVqMc7Pz7c4JpFIkJOT43AAvNobkpZzLTTdQYslrpajXhDwzfZs6AUdstbuNxoTRSoXQCYPREXFXqhUKsyZM8fu93C1PmOr0xQymczkkEgkqKqqQmFhIZMAeLU3JC3nWmi6gxZLXC3Hns4WXLn0MyJi55s4xEXGGfySDf7I9uNq0xRWi/Hy5cuNR1paGgYHB1FVVQWVSoX6+nomAfBqb0hazrXQdActlrhajtpbBrc2QTCd6hh6oKZz8MGaq7m2jfgA7/bt29i6dSuSk5Oh0+lw6NAh5ObmIigoiEkAvNobkpZzLTTdQYslrpZjyIQpAICOZjV0Dx8MG2tvOgjAsHO1I7haN4XVOeO8vDzU1dUhLS0NarUacrmceQC82huSlnMtNN1BC+DTDtURrXOnv0Xf34Ydrvu1N6HTPcDJI7sM+QVNQNTsxQCA0OcjEBH7Oi60/oCiT1MxMz4ZXt5SdGlO4lL7MURHRyMxMdHmmM3xVFlolpSUwNvbGwUFBcM2HyULTdJytdh41eLVDtVerbbGKvRcHD7X21C9HQAwceosYzEGgJSVn6G5fiY6mtVoqP4cer2AgGcVmPPGe9i58X0GrW2jN/0gCAK2bduGyspKDAwMIDY2FuvXr4dCoRj5jy1AFppuoDWkR1ritAC2nyULq1Br3hT2wPqaWfKhEMuyuY5rzFvSZPNrGw6K69rYs2cPysvLUVhYiNDQUGzatAltbW2orq62uFPKSDh90QdBEMRooNcLNh9iKS8vR3Z2NsLCwiCXy5Gbm4vu7m60tLTYHS9ZaBIE8VQi5o6/pizG7O7W/v7+8Pcf3lWi1WoRFxeHiooKREVFGc8vXLgQ6enpyMzMtCtep6/AIwiCGA3E7BayY8cOs4vcVq1ahdWrVw87d+eOobXv8SLt5+dnHLMHKsYEQbg9WVlZSElJMTn/eMEFAF9fXwCGO+RH0Wq1xjF7oGJMEITbY246whJ+fn5QKBTo6Ogw9kJrtVr09PQgMjLS7hjoAR5BEIRIMjIyUFRUhO7ubvT392Pz5s2YPHkylEql3Zp0Z0wQBCGS7OxsaLVaLF26FAMDA1AqlSgoKLC7rQ2gbgqCIAguoGkKgiAIDqBiTBAEwQFUjAmCIDiAijFBEAQHUDEmCILgACrGBEEQHEDFmCAIggOoGBMEQXAAFWOCIAgO+Bfz8SlJHXO1VQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lvVQJ_0MqtOT"
      },
      "source": [
        "# 5 Model 9 : CamemBERT to test domain shift\n",
        "\n",
        "For **Model 9**, we also fine-tuned the pre-trained CamemBERT model but this time, we trained the model on the Onisep dataset and tested it on the Pôle Emploi dataset to see if there is a **domain shift** between the 2 datasets. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LFPGYi2yqtOV",
        "outputId": "3efc9071-fa73-4da1-87fe-dc798784015f"
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Thu Apr 15 12:30:46 2021       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 460.67       Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   70C    P0    32W /  70W |   1356MiB / 15109MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 462
        },
        "id": "V7GcQ62rqtOY",
        "outputId": "f43e557a-e9a6-4ac6-879d-cf190b0eb977"
      },
      "source": [
        "# We keep only the relevant columns : we want to predict the major code (grand domaine)\n",
        "# using the description\n",
        "y_variable = \"major_code\"\n",
        "data_bert= data[[\"description\", \"table\", y_variable]].copy() #NEW : we keep table\n",
        "\n",
        "# we change the grades to 0-numbering because it is the classical way to do to perform the models\n",
        "tag_to_ix = {}\n",
        "i=0\n",
        "for code in data_bert[y_variable].unique():\n",
        "  tag_to_ix[code]=i\n",
        "  i += 1\n",
        "print(tag_to_ix)\n",
        "data_bert['label'] = data_bert[y_variable].replace(tag_to_ix)\n",
        "data_bert"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'A': 0, 'B': 1, 'C': 2, 'D': 3, 'E': 4, 'F': 5, 'G': 6, 'H': 7, 'I': 8, 'J': 9, 'K': 10, 'L': 11, 'M': 12, 'N': 13}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>description</th>\n",
              "      <th>table</th>\n",
              "      <th>major_code</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Réalise des travaux mécanisés agricoles, sylvi...</td>\n",
              "      <td>rome</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Tracteur, moissonneuse-batteuse, ensileuse, ép...</td>\n",
              "      <td>onisep</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Depuis la cabine de sa machine, le conducteur ...</td>\n",
              "      <td>onisep</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Réalise des opérations de coupe et d''entretie...</td>\n",
              "      <td>rome</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Avec ses cordes de rappel, son harnais, ses mo...</td>\n",
              "      <td>onisep</td>\n",
              "      <td>A</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1242</th>\n",
              "      <td>Surveille et coordonne les mouvements des trai...</td>\n",
              "      <td>rome</td>\n",
              "      <td>N</td>\n",
              "      <td>13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1243</th>\n",
              "      <td>Le technicien de la circulation ferroviaire SN...</td>\n",
              "      <td>onisep</td>\n",
              "      <td>N</td>\n",
              "      <td>13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1244</th>\n",
              "      <td>Réalise des opérations de mise en fonction, de...</td>\n",
              "      <td>rome</td>\n",
              "      <td>N</td>\n",
              "      <td>13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1245</th>\n",
              "      <td>Sur les pistes de ski, une panne d'une télécab...</td>\n",
              "      <td>onisep</td>\n",
              "      <td>N</td>\n",
              "      <td>13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1246</th>\n",
              "      <td>Réalise des opérations de formation de trains ...</td>\n",
              "      <td>rome</td>\n",
              "      <td>N</td>\n",
              "      <td>13</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1247 rows × 4 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                            description  ... label\n",
              "0     Réalise des travaux mécanisés agricoles, sylvi...  ...     0\n",
              "1     Tracteur, moissonneuse-batteuse, ensileuse, ép...  ...     0\n",
              "2     Depuis la cabine de sa machine, le conducteur ...  ...     0\n",
              "3     Réalise des opérations de coupe et d''entretie...  ...     0\n",
              "4     Avec ses cordes de rappel, son harnais, ses mo...  ...     0\n",
              "...                                                 ...  ...   ...\n",
              "1242  Surveille et coordonne les mouvements des trai...  ...    13\n",
              "1243  Le technicien de la circulation ferroviaire SN...  ...    13\n",
              "1244  Réalise des opérations de mise en fonction, de...  ...    13\n",
              "1245  Sur les pistes de ski, une panne d'une télécab...  ...    13\n",
              "1246  Réalise des opérations de formation de trains ...  ...    13\n",
              "\n",
              "[1247 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 959
        },
        "id": "w9hGgGBwteve",
        "outputId": "7a56ba67-a23a-4108-8395-e705b220297f"
      },
      "source": [
        "# create train : ONISEP data\n",
        "# create test : ROME data\n",
        "data_bert['data_type'] = 'not_set'\n",
        "# NEW way of defining train and test \n",
        "X_train = (data_bert['table']==\"onisep\")[data_bert['table']==\"onisep\"].index.values\n",
        "X_val = (data_bert['table']==\"rome\")[data_bert['table']==\"rome\"].index.values\n",
        "y_train = data_bert.loc[X_train, 'label'].values\n",
        "y_val = data_bert.loc[X_val, 'label'].values\n",
        "data_bert.loc[X_train, 'data_type'] = 'train'\n",
        "data_bert.loc[X_val, 'data_type'] = 'val'\n",
        "data_bert.groupby(['major_code', 'label', 'data_type']).count()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>description</th>\n",
              "      <th>table</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>major_code</th>\n",
              "      <th>label</th>\n",
              "      <th>data_type</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">A</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">0</th>\n",
              "      <th>train</th>\n",
              "      <td>44</td>\n",
              "      <td>44</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>30</td>\n",
              "      <td>30</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">B</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">1</th>\n",
              "      <th>train</th>\n",
              "      <td>21</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>19</td>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">C</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">2</th>\n",
              "      <th>train</th>\n",
              "      <td>19</td>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>25</td>\n",
              "      <td>25</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">D</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
              "      <th>train</th>\n",
              "      <td>35</td>\n",
              "      <td>35</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>39</td>\n",
              "      <td>39</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">E</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">4</th>\n",
              "      <th>train</th>\n",
              "      <td>43</td>\n",
              "      <td>43</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>23</td>\n",
              "      <td>23</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">F</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">5</th>\n",
              "      <th>train</th>\n",
              "      <td>58</td>\n",
              "      <td>58</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>38</td>\n",
              "      <td>38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">G</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">6</th>\n",
              "      <th>train</th>\n",
              "      <td>34</td>\n",
              "      <td>34</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>30</td>\n",
              "      <td>30</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">H</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">7</th>\n",
              "      <th>train</th>\n",
              "      <td>122</td>\n",
              "      <td>122</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>94</td>\n",
              "      <td>94</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">I</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">8</th>\n",
              "      <th>train</th>\n",
              "      <td>31</td>\n",
              "      <td>31</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>28</td>\n",
              "      <td>28</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">J</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">9</th>\n",
              "      <th>train</th>\n",
              "      <td>38</td>\n",
              "      <td>38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>32</td>\n",
              "      <td>32</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">K</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">10</th>\n",
              "      <th>train</th>\n",
              "      <td>132</td>\n",
              "      <td>132</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>71</td>\n",
              "      <td>71</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">L</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">11</th>\n",
              "      <th>train</th>\n",
              "      <td>33</td>\n",
              "      <td>33</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>22</td>\n",
              "      <td>22</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">M</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">12</th>\n",
              "      <th>train</th>\n",
              "      <td>78</td>\n",
              "      <td>78</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>44</td>\n",
              "      <td>44</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">N</th>\n",
              "      <th rowspan=\"2\" valign=\"top\">13</th>\n",
              "      <th>train</th>\n",
              "      <td>27</td>\n",
              "      <td>27</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>val</th>\n",
              "      <td>37</td>\n",
              "      <td>37</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                            description  table\n",
              "major_code label data_type                    \n",
              "A          0     train               44     44\n",
              "                 val                 30     30\n",
              "B          1     train               21     21\n",
              "                 val                 19     19\n",
              "C          2     train               19     19\n",
              "                 val                 25     25\n",
              "D          3     train               35     35\n",
              "                 val                 39     39\n",
              "E          4     train               43     43\n",
              "                 val                 23     23\n",
              "F          5     train               58     58\n",
              "                 val                 38     38\n",
              "G          6     train               34     34\n",
              "                 val                 30     30\n",
              "H          7     train              122    122\n",
              "                 val                 94     94\n",
              "I          8     train               31     31\n",
              "                 val                 28     28\n",
              "J          9     train               38     38\n",
              "                 val                 32     32\n",
              "K          10    train              132    132\n",
              "                 val                 71     71\n",
              "L          11    train               33     33\n",
              "                 val                 22     22\n",
              "M          12    train               78     78\n",
              "                 val                 44     44\n",
              "N          13    train               27     27\n",
              "                 val                 37     37"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "syx-oJ0BqtOb"
      },
      "source": [
        "In the following code we format the data so that it is easily usable in the model. In particular we specify a maximum length for the sentence (= maximum number of tokens) to 256. Indeed, we notice that to increase its value dramatically increases the time computation without any significant gain in the quality of the model: the first tokens are enough to predict the label. This value is higher than the one used in the previous section because CAMEMBERT divides the sentence in much more tokens (especially due to the byte pair encoding for unknown words).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nzcICiZcqtOd"
      },
      "source": [
        "tokenizer = CamembertTokenizer.from_pretrained(\"camembert-base\", \n",
        "                                          do_lower_case=True)\n",
        "                                          \n",
        "encoded_data_train = tokenizer.batch_encode_plus(\n",
        "    data_bert[data_bert.data_type=='train'].description.values, \n",
        "    add_special_tokens=True, \n",
        "    return_attention_mask=True, \n",
        "    pad_to_max_length=True, \n",
        "    max_length=256, \n",
        "    return_tensors='pt'\n",
        ")\n",
        "\n",
        "encoded_data_val = tokenizer.batch_encode_plus(\n",
        "    data_bert[data_bert.data_type=='val'].description.values, \n",
        "    add_special_tokens=True, \n",
        "    return_attention_mask=True, \n",
        "    pad_to_max_length=True, \n",
        "    max_length=256, \n",
        "    return_tensors='pt'\n",
        ")\n",
        "\n",
        "\n",
        "input_ids_train = encoded_data_train['input_ids']\n",
        "attention_masks_train = encoded_data_train['attention_mask']\n",
        "labels_train = torch.tensor(data_bert[data_bert.data_type=='train'].label.values)\n",
        "\n",
        "input_ids_val = encoded_data_val['input_ids']\n",
        "attention_masks_val = encoded_data_val['attention_mask']\n",
        "labels_val = torch.tensor(data_bert[data_bert.data_type=='val'].label.values)\n",
        "\n",
        "dataset_train = TensorDataset(input_ids_train, attention_masks_train, labels_train)\n",
        "dataset_val = TensorDataset(input_ids_val, attention_masks_val, labels_val)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jXa-2euPqtOg"
      },
      "source": [
        "#load the pretrained camembert model\n",
        "camembert = CamembertForSequenceClassification.from_pretrained(\"camembert-base\",\n",
        "                                                      num_labels=len(tag_to_ix),\n",
        "                                                      output_attentions=False,\n",
        "                                                      output_hidden_states=False)\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "camembert.to(device)\n",
        "optimizer = AdamW(camembert.parameters(),\n",
        "                  lr=1e-5, \n",
        "                  eps=1e-8)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "216Cv0AQqtOi"
      },
      "source": [
        "# 5 epochs is enough to have good performance             \n",
        "epochs = 5\n",
        "batch_size = 3\n",
        "\n",
        "dataloader_train = DataLoader(dataset_train, \n",
        "                              sampler=RandomSampler(dataset_train), \n",
        "                              batch_size=batch_size)\n",
        "\n",
        "dataloader_validation = DataLoader(dataset_val, \n",
        "                                   sampler=SequentialSampler(dataset_val), \n",
        "                                   batch_size=batch_size)\n",
        "# check documentation, mentionned here: https://huggingface.co/transformers/training.html\n",
        "scheduler = get_linear_schedule_with_warmup(optimizer, \n",
        "                                            num_warmup_steps=0,\n",
        "                                            num_training_steps=len(dataloader_train)*epochs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z5HKvEBBqtOl"
      },
      "source": [
        "# The model by epoch is stored in a folder data_volum\n",
        "!mkdir data_volume"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e_XDsz8sqtOo"
      },
      "source": [
        "random.seed(seed_val)\n",
        "np.random.seed(seed_val)\n",
        "torch.manual_seed(seed_val)\n",
        "torch.cuda.manual_seed_all(seed_val)\n",
        "    \n",
        "for epoch in tqdm(range(1, epochs+1)):\n",
        "    \n",
        "    camembert.train()\n",
        "    \n",
        "    loss_train_total = 0\n",
        "\n",
        "    progress_bar = tqdm(dataloader_train, desc='Epoch {:1d}'.format(epoch), leave=False, disable=False)\n",
        "    for batch in progress_bar:\n",
        "\n",
        "        camembert.zero_grad()\n",
        "        \n",
        "        batch = tuple(b.to(device) for b in batch)\n",
        "        \n",
        "        inputs = {'input_ids':      batch[0],\n",
        "                  'attention_mask': batch[1],\n",
        "                  'labels':         batch[2],\n",
        "                 }       \n",
        "\n",
        "        outputs = camembert(**inputs)\n",
        "        \n",
        "        loss = outputs[0]\n",
        "        loss_train_total += loss.item()\n",
        "        loss.backward()\n",
        "\n",
        "        torch.nn.utils.clip_grad_norm_(camembert.parameters(), 1.0)\n",
        "\n",
        "        optimizer.step()\n",
        "        scheduler.step()\n",
        "        \n",
        "        progress_bar.set_postfix({'training_loss': '{:.3f}'.format(loss.item()/len(batch))})\n",
        "         \n",
        "        \n",
        "    torch.save(camembert.state_dict(), f'data_volume/finetuned_CAMEMBERT_epoch_{epoch}.model')\n",
        "        \n",
        "    tqdm.write(f'\\nEpoch {epoch}')\n",
        "    \n",
        "    loss_train_avg = loss_train_total/len(dataloader_train)            \n",
        "    tqdm.write(f'Training loss: {loss_train_avg}')\n",
        "    \n",
        "    val_loss, predictions, true_vals = evaluate(dataloader_validation)\n",
        "    val_f1 = f1_score_func(predictions, true_vals)\n",
        "    tqdm.write(f'Validation loss: {val_loss}')\n",
        "    tqdm.write(f'F1 Score (Weighted): {val_f1}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0da1DybqqtOu"
      },
      "source": [
        "pred_vs_true_m9 = matrix_pred_bert(dataloader_validation)\n",
        "y_true = pred_vs_true_m9.label\n",
        "y_pred = pred_vs_true_m9.y_pred"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 726
        },
        "id": "DghkkzmG3S2l",
        "outputId": "208a5289-06b7-4e46-af85-076d3d1a0415"
      },
      "source": [
        "print( classification_report(y_true, y_pred ) )\n",
        "cm = confusion_matrix(y_true, y_pred , labels = key_list) \n",
        "df_cm = pd.DataFrame(cm, index=key_list, columns=key_list)\n",
        "# config plot sizes\n",
        "sn.set(font_scale=1.2)\n",
        "sn.heatmap(df_cm, annot=True, annot_kws={\"size\": 18}, cmap='coolwarm', linewidth=0.5, fmt=\"\")\n",
        "plt.tick_params(axis='both', which='major', labelsize=10, labelbottom = False, bottom=False, top = False, labeltop=True)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           A       0.86      0.83      0.85        30\n",
            "           B       0.00      0.00      0.00        19\n",
            "           C       1.00      0.16      0.28        25\n",
            "           D       0.87      0.69      0.77        39\n",
            "           E       0.56      0.22      0.31        23\n",
            "           F       0.74      0.68      0.71        38\n",
            "           G       0.96      0.77      0.85        30\n",
            "           H       0.55      0.96      0.70        94\n",
            "           I       0.00      0.00      0.00        28\n",
            "           J       0.95      0.56      0.71        32\n",
            "           K       0.50      0.89      0.64        71\n",
            "           L       0.85      0.77      0.81        22\n",
            "           M       0.55      0.59      0.57        44\n",
            "           N       0.96      0.65      0.77        37\n",
            "\n",
            "    accuracy                           0.65       532\n",
            "   macro avg       0.67      0.56      0.57       532\n",
            "weighted avg       0.66      0.65      0.62       532\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAEDCAYAAADgNmW7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd3hUVeL+PzOTSe8JENLoJEIoAZQOBpBOaCIiILIromJ33V2QtawKqF9ddkHBBf0BAiJ9pSkoSAs1UoOhht4SUob0ZGZ+f4wJhCncmdzJnIT7eZ77QG555z13Tk7OPffc96qMRqMRBQUFBQWXona1AQUFBQUFpTFWUFBQEAKlMVZQUFAQAKUxVlBQUBAApTFWUFBQEAClMVZQUFAQAKEa459//pmYmBjOnj1bKZ2HHnqIwYMHk5iYyNChQ/ntt98qpZeens7rr79Or169GDZsGBMmTCAtLc1hXwMGDCAxMZFvvvkGg8HgsK8yvbLlv//9r2xaly9fdlgrIyODN998k549ezJs2DBGjhzJli1b7NaJj4+v8PPq1av55z//6bAva7qu1rv7+O3bt9OnTx+uXLniMj9lxMTE8Je//KX859LSUjp06MDEiRMd0poxY0b5z19//TWzZs2SxWdNwc3VBu5m/fr1tG3blg0bNvDKK684rOPp6cn//vc/AHbu3Mnnn3/O4sWLHdIyGo289NJLDBkyhH/9618ApKamcuvWLRo0aOCwr1u3bvHmm2+Sm5vrcFnv1qsscmkZjUYmTZrEkCFD+OyzzwC4cuUKW7durbR2TWfPnj18+OGHfP3110RERLjaDt7e3pw+fZrCwkI8PT3ZvXs3derUcUjL3d2dzZs389xzzxEcHCyz05qBMD3jvLw8kpOT+eijj9iwYYNsurm5ufj7+zt8/N69e3Fzc2PUqFHl62JjY2nXrl2lfIWEhPDBBx+wZMkSatJzN3v37kWr1VY4XxEREYwdO9aFrsTnwIEDTJ06lblz5xIdHe1qO+V0796dX3/9FYANGzYwYMAAh3Tc3NwYOXIkCxculNFdzUKYnvEvv/xC165dadCgAUFBQRw/fpy4uDiHtAoLCxk8eDBFRUWkp6dXqgKcPn2a5s2bO3y8LaKiotDr9dy6dYvQ0FC7jy8rZxkTJ06kf//+Dnm5WysyMpIvvvjCIZ3Tp0/TrFkzh4615QkgJyeHHj16yKItEsXFxUyaNIlFixbRqFEjV9upQP/+/fnyyy9JSEjg5MmTDB8+nOTkZIe0Ro8eTWJiIs8++6zMLmsGwjTGGzZs4OmnnwZMFWDDhg0ON8Z3X3IfOnSIv/3tb6xfvx6VSiWbXxEQcZjiXt5//32Sk5PRarWsWrWqUp5Wr17N8ePH5bboctzc3IiPj2flypVMnTrV1XYqEBsby+XLl1m/fj3du3evlJavry+DBw9m0aJFeHp6yuSw5iDEMEV2djZ79+5l6tSp9OjRg6+//ppNmzbJcvkeHx9PVlYWmZmZDh3fpEkTUlJSKu3DEpcuXUKj0RASEuIUfVfQpEkTTpw4Uf7zu+++y4IFC8jKynKhK7FRq9XMnDmTY8eOMXfuXFfbMaNHjx588sknDg9R3M24ceNYtWoVBQUFMjirWQjRGP/0008MHjyYbdu2sXXrVrZv305kZCQHDx6stPbZs2fR6/UEBgY6dHyHDh0oLi7m+++/L1+XmppaaW+ZmZm8++67jB49ukb12Dt06EBRURFLly4tX1dYWOhCR9UDLy8vvvrqK9atW8eKFStcbacCjz/+OJMmTSImJqbSWoGBgfTt25eVK1fK4KxmIcQwxfr165kwYUKFdb1792b9+vU8/PDDduvdPdZoNBr5+OOP0Wg0DnlTqVTMnj2badOmMW/ePDw8PIiIiGDKlCkO+yotLUWj0TB48GDGjx/vkK+79cro2rVrhalIrkClUvHFF18wffp05s+fT3BwMF5eXi735SxKS0txd3eXRSswMJD58+czevRogoOD6dmzp0M6BQUFdOvWrfzn8ePHV6qehYWFlQ8hysGf/vQnlixZIpteTUGlRGgqKDhOamoqU6dOVXp6CpVGiJ6xgkJ15LvvvuPbb7916CpJQeFelJ6xgoKCggAIcQNPQUFB4UFHaYwVFBQUBEBpjBUUFBQEQGmMFRQUFARAaYwVFBQUBEBpjBUUFBQEQGmMFRQUFARAaYwVFBQUBKDKnsBLS0vjhx9+YPfu3Vy8eJGioiKio6Pp27cv48aNw9vbu3zfWbNmMXv2bIs6f/3rX/nzn/9cVbYVFBSqKRu00oONBpScdKITaVRZY7xq1SoWLFpCo7getOoxCLXGjcun9zFz5kwWL9/EqDeW4+Zuyjg9eMZ0TPehk/HyDaqgc93tTsZxl0HbzT7n1QmNGJEYyfakdPYmZ1IvyofHB4ZzJCWH1/5xlHufN9y1rnuN1yrTc7aWo+V8bLR5WHl8cz9GDQ4j9Uwe6Zkl6PVGGkR50ad7CPkFep6f8jvZutIKx2xZ0lZWXyDvd5nwxL4K6yc8FcWQPnVIOpjFidO5lOqNxDf3J6FTCGfP5/Hi2ykUl1QU27a8vWy+yryJqvWgUWWNcZ8+fdA2noiHl1/5ulZdRhFYqx77N8/l+N6VtO42psIxjVr2IiAkUvJnNIj2ZvjACH5NSmfq9DuZutduFPD6xCb06labLdtvKlpO0JJb71DKbQ6l3DZbfyz1Nv94tRG9u4WwfP2NKvclp9b2vZksXXOVvAJ9+bp1W25y+VohY4dH0L9Hbdb+VL3LKHcdsweVtnpF01bZmHGLFi0qNMRlxLQxvSYo49opi8cVFeRi0Jda3HYvvbrVRq1Wsfx/Fd+su+6naxQU6un9aG3JfhUt+7ScoWeJGxnFAPj6SI9EFfWcnTqXV6EhLmNb0i0AGkR7ucSXqFr2ovHSSF5EwOWpbbezrwPg7Wf+DrjFMxIpLspDpdYQFt2S9n1foEEz65cvsU380OuN/H5KV2F9cYmR0+dyiW1i/sdA0ZJHyxl6AFqtCi8PNe7uaupFePLsk6Yrpf2Hdfc50jm+nFHGe6kVYspHzsoucYkvUbXsRe1Ww3vGBw8e5P3335flww0GPft+nINa7UZs24Hl6z28/GjRaSSPPv4PEid8SZdBb6DLusLaryaSsm+1Vb3QYHdydCWUlJoPQmVkFhEU4I6bxC9I0bJPyxl6AP0fDWXVV635blZLZvy9KT7eGqZ/kcbxk7ku8eWMMt6NWgVjh0dQWmrgl123JB8nahmdfb5sodKqJC8iIKlnfOLECdatW8dPP/1EREQEvXv3luXDt6+exrXzh+g88A2C6zQsX98m4RmzfZt3GM6i6YPYvno6eW/2wcfHx2wfTw8NJaUGi59VXGz4Yx81uaXml4aKVuW0nKEHsDs5m4tXC/HyVNO4vjcd2wQS4GffBZ3I5+xeJj1Tj7gYP+YtvcSla9JfVyVqGZ19vmxR3XrGVmt1WloaGzZsYP369QQFBdG/f3+MRiPffvutLB+ctGEmh3cspkWnkTzSe+J99/fyCaJl5yfZu2kWhw4dokuXLmb7FBbpCfK0/Aocd3f1H/tYrhiKVuW0nKEHkJFZQkam6XI9KTmHnfuzmf1BLB4eapb9cL3KfTmjjGWMHxnJsH5hrNtyg6Vrr9p1rKhldOb5uh+i9HilYnWYol+/fuzdu5evvvqK7777jrFjx6JWy3O/b8/GWez7aQ7N2w+j50jpQx7+wREAVt80nJFZTIC/Fq2Fv4ihwR5k5RRTauFySdGqvJYz9CyRdqmAsxfyGdSrlkt8OauM40ZE8PTwCDZtu8nn887bfbyoZayKOmENtZtK8iICVlvX2bNnU6tWLZ5++mmmTp3Knj17kOOlIHs2zmLvj7Np9shQHhv1kV1vRs5OPw9AaKj5zT6A1NO30WhUPNTUv8J6d62KJg19OXlG+jijomWfljP0rOGuVeNnx2wKkc8ZmBriZ0ZE8uOv6Xw6N83u4+X2JaqWvWjc1ZIXEbDqolevXvzrX/9i06ZNtG/fnoULF5a/Xn7Xrl0OfdjeTbPZ++NsHnp4ML2fmobKQk/boC+lqMB8funtrGsc3bUMT59A4uPjLer/sjMdg8HIE4MjKqwf1KcuXp4aNv8qbc6momW/ltx6QQGWR9BaNfOlfpQXqWfyXOJL7nP29HBTQ7x5ezqfzDln8QGIqvYlqpa9qNQqyYsI3PdOiLe3N4MGDWLQoEHk5OTw448/Mm/ePItjtrZYsmQJezbNwi8onOiYTqQmr6v4OX6h1IvtTHFRPt+835NGLXsRXKchnt4BZN1M4/ieFRQX5dN/3Gd4enpa/IxzF/JYveEqjw+K4KPJzdiTnEn9SG8eHxTBoWPZdk0uV7Ts05Jb75Xx0YQEaTmccpsbGcW4a9U0aeDNox2DKCjQ89XSyy7xJafWkD51GD8ykuvpRSQf09GzS0iF7VnZJSQfkzaFT9Qyyl3H7EGlEaPHKxW7bksHBAQwcuRIRo4cafcHHTt2DIDbWVf5afHfzLZHNn6EerGdcdN60qRVb65dOMrZoz9TUpSPp28QUU078XCvZwmr19Lm5/xn/hmu3ywksU9dOj4cQo6uhJXrr/L1kjS7ex2Kln1acupt25PFY12D6dklhEA/N4yYHvjY8EsGyzdcJ/2W9Dm4cvqSUyumkWlGUFgtDya/1Mhs++EUneTGWE5fImvZg1ojRo9XKlX6dui5P8mj83wf07/WcgjswVYGQU3RKtMTVctSNoUj2MqmsBdnfJf3ZlM4gq1sCkcQuV5Uln0d20vet/2eyn83lcXlT+ApKCgoOANRbsxJpUp7xgoKCgpVRXJCZ8n7tt2224lOpKH0jBUUFGokosySkEqVNsYiZ83KqdVtqPnUvx1rupBzu4RTZ3OJaeRLXoGekRMPWjwfZfvL5avMm6h5xgB5BQYu3dQTVUdDYRG8+7X59MZ7iail5q9P+VKiBxXg4a6yeO7Btee/rIz3jo07mtksl68yb6JqVZbqdgPP5T1jUbNT5c5hHfn8Aa7dKAJgwb/j8fJ0LLZP5DI6qvfu1zpu5Zh+I6c87YuHhMdYVSp46jEvTpwvxdNdRXQd2+dTxPP/IGQ2uzTPuJr1jF0+wi1qdqrcOaxlDUFlEbmMjuqVNcT28Gi8O2EhGpZvLZC0v4jn3xo1KbPZlXnGKrVa8iICLu8Zi5qd6soc1qryVR3yjC0R5KdiYCdPNu0pJOt21d5/VjKbq8/vkUYrRiMrFZe7FTU71ZU5rFXlqzrkGVviyV5eZOQY2JpcXGkte1Eym6vP75EzH4fOyMjgzTffpGPHjrRr144nn3ySAwcOlG9PSkoiMTGRVq1a0adPHzZu3HhfTbt7xpmZmQQFBdkV8GMLUbNTXZnDaguRy1gV56xtjJaH6rvxr2V5GFwwKVPJbK4+v0fOHH54//33yc7OZsOGDQQEBLBw4UKee+45tm/fjk6n44UXXmDq1KkMHjyY3bt389prrxEREUGrVq2satp0e/jwYcaOHctLL73EiRMnGDhwIIMGDaJTp07s2LFDlkIVFunRulm24Uh2qohaciJyGZ19zrw9VQx/1JM9x0tIu1a1fwTLcEYZMzJLOJRym6TkHBatusYnc8/z7KgInkwMc4kvUbXsxZk94wsXLtC3b1+Cg4PRaDSMHDmS/Px8Ll68yJo1a2jatCkjRozA3d2dhIQEEhISWLZsmU1Nm43xP//5TyZOnMiAAQMYN24cH374Ibt372bx4sV8/vnndhfAEqJmp7oyh7WqfFW3PON+HTzw0KpIOlZMaKC6fNG6YZrbBtQOsRxkLhdKZnP1+T2ypzHW6XRcvnzZbNHpLI/bT5gwgc2bN5Oenk5JSQlLliyhfv36NG3alNTUVOLi4irsHxcXR2pqqk2/NhtjvV5Ply5d6NevH6GhobRu3RqARo3MQ00cRdTsVFfmsFaVr+qWZxzsr8bDXcVbT/ny3p/8ypf6dd3Kp8N98k7zSn3G/VAym6vP75HaTSN5WbhwIT179jRbFi5caFE7Pj4eNzc3unTpQqtWrViwYAEzZszA3d2d3Nxc/P0rltff35/cXNtltdkY3/1mj3tjK+UaMxY1O9WVOaxV5UvkPGNL/HygiPnr8syWaxl6iv/oXc3+xrFwdqkomc3V5/dIrVFJXsaNG8cvv/xitowbN85M12Aw8MwzzxAWFsb+/fs5evQo//znP5kwYQInT57E19eX27crzh/X6XT4+vra9GvzLkFqaipt2rTBaDRSVFREmzZtADAajRQXy3MnW9TsVLlzWHt3r0VYLdMftEB/LVo3NU8/HgXA9fRCNm9Pr/ZldFTv4Ye0BPub/vD7eqlw06jo094DgEydgQO/m+IyrY0Td29tJPiPjsjBI9kW9xHx/D8Imc0uzTO2YyzY39/frDdrjZycHC5dusQXX3xBQEAAYHoZR1RUFLt37yY2NpadO3dWOCYlJYXY2FibujYb499//12SucoianaqnFoDeoURHxdQYd2zo+sBcOh4juTGQG5fIuQZd4pzp0lUxao4qLOp4Tx9qbS8Ma4MIp7/ByGzWW4te3DWbIqgoCAaNWrEkiVL+Otf/4q3tzfbtm3j9OnTNG/enPDwcObPn8+qVatITEwkKSmJbdu2WR3yKMPlD30AGAywbO1llq2V3hOoblqv/uNYpTXKELWMjur9e4X0y3Fbx89+I8DqPiKe/x37stixz/LLdR1B1Hohdx2TijMfh/7yyy/55JNPeOyxxygqKiIiIoJ3332X9u1NGcpz5sxh+vTpvP/++4SFhTFt2jSb09pAkMZYQUFBQW6c2RjXr1+fL7/80ur2Tp06sW7dOqvbLaHkGSsoKNRILr04XPK+UV+ucqITaQgRoalSwYjECAb3DSestifZOcVs25XO/CXnLU4ItxV76ahWv2eO3te/h7uKOR82pW5tD374OYM5i69W2L5pQUvZfcmhVaZnKV5y9LBImjbyJaahL+Fhnly7WWgzWhJM8ZJyfpeiaoG83+VTfze/TPf3VfN4L3/iYz0J8NWQnavnYEoBK7foyC807yctnREpm68ybwdSCqkfrqVWkIaMbD1//fcti/s2a6il7UOe1K/rRmQdN7RuKj5ekMXJC6ax7W/erS2rr8oiSgCQVIQYpnjl2Tt5p8vWXDLlnQ6KoElDX6t5p1WhdS9jh4bZ/ZiqM3zJqTVxbP3ynF97UsKqwltN1/L3UfPBpNoE+Wn4ZX8el66XEBWmpVd7X2IbePDenHSKS6Qbc9TXQw3cuXCtFC9P25f1HVp40qGFJ1dulnI1vZR6dbVO9VVpZJp+W1W4vDGuLtmpjep5MaR3KF8vv8Zzo8LtOlbkMsqV8yu3twdBa3CCH7WC3Jj13S32HLkTBXrqQhEvjwqhf1df1m69f8h+ZX298mkGAP98IRhPd+sN2OqteSxaf5tSPfTp6CWpMVbyjKXj8n58dchOVavg1fERHDx2m93JOXYfL3IZ5cr5ldvbg6DVrJEHRcWGCg0xwN6jBRSXGOne1sclvqyRfduAvXk+Sp6xdGy6uHDhAsnJ5q9RT05O5uLFi7IYqA7ZqUP7hBJV15MvF1+5/85O9iVqzjKIW05RtbQaFSWl5uuNRpNenRA3/LylNRSi1gtX+nJmUJAzsPlNT5s2zeIjfL6+vkybNk0WA6Jnp9YJ1TJmaBhL/3eDmxmOPXwgehnlQtRyiqp1+WYJvt5qs8v9enW1+P7RCIcEShs2ErVeuNKXPdkUImCzMc7IyCAmJsZsfUxMDFeuONZLvBepeadVrVXGy+MiuXaziNU/SX9Cy5m+nFFGuRC1nKJqbdqVi8Fg5JWngmkd40lIgIZWTT15+ang8iQzKe8DlNuXnLjSV3XrGdu8gXdv2MXdFBYWymKgsEhPkKfl2ENHslPl0gJI6BhIfHNf/jr9LPpKxOeKXEY5EbWcomqdPF/MrO8yeXpQIH8dHwqAXm9k24E8LvuW8kicF/lF0qYaiFovXOpLkLFgqdhsjOPi4li+fDlPPPFEhfUrVqygeXN5ogozMoupH+WD1k1ldinjSHaqXFpaNxXPjQrnwNHbZOaUUre2qUKFBpkuKX28NdSt7Y4ut5S8fNuVSdQyyo2o5RRVC2DfsQL2Hy8gOkyLp4eKa+ml6PIMfDCpNqV6IzduWRhUrgJfcuFKX3IlS1YVNv90TJkyhdWrVzN27FhmzJjBjBkzGDNmDCtXruTtt9+WxYCo2anu7ioC/d1o39qfbz6JLV8+mWzKcu7ZKYhvPomlb7fgKvUlas4yiFtOUbXKMBrhwrUSTp4vRpdnIMBXTb1wLanniiTPMxa1XrjSV42aTREaGsqyZcuYNGkSERERREREMGnSJL7//ntq1ZL+FgJbiJqdWlhk4KPZF8yW2QtNT1EdOKrjo9kX2Hvo/m/wFbWMciNqOUXVsoRKBeMSA1GrYO02aXOMq8KXo7jSl8pNI3kRAUkPfXTo0IEOHTo4xYCo2al6Pew6aD6nuHaoaZji2s1ii9ud7UvUnGWRyymqloe7ig8m1eZgSgE3s0rx9lTTqZU3DSPd+f7HHE6ckz4HvDK+Bnb1BsDPW42b5s7Pt3IM7Dl6595QZG0NrWNMOdNNoky/B51aedIk2vrDH9Ulz1gEXP4EHtSM7NSq9CVqzrLc3mq6VqneyMVrJXRq7U2gn4biEiPnLhcz4+t0jp62/2EcR30N6+Fr8efU88UVGuN6dbVm+3aN93Kar8qiUokx/CAVIRrj6pSdejOjRFKo0L2IWkY5c35B3HKKqKXXw+xlmZX2U4ajvv70vrTe6e4jhew+Yn0W1TfvWn6azlV5xlSznrESoamgoFAjyf74Jcn7Bv5tthOdSEOInnFNIi0tjR9++IHdu3dz8eJFioqKiI6Opm/fvowbNw5vb+/yfS09UHM3r732Gi+88IKzLSso1EiUMWMbJDyxz2zdhKeiGNKnDkkHszhxOpdSvZH45v4kdArh7Pk8Xnw7xWx6z7blplebWMpOfXXCnbi+vcmZpri+geEcScmxGNdnK7fWEa1Vq1axcNESGrfsQdveg9Bo3Lhwch8zZ85kyYpNPP235WjdTTfMBo7/xOJ52rV+NtnpF0lISJDFV5k3ObUsZSPvWNOlPI4zppEveQV6SdnIj402zz+Jb+7HqMFhpJ7JIz2zBL3eSIMoL/p0DyG/QM/zU34nW1dxDu6WJW2t5hk7Wi/kKOeONV0A8/rv6rpfVk45fJV5k7NeVBaVRoxZElJxec94+95Mlq65Sl7BnUfc1m25yeVrhYwdHkH/HrVZ+5O06S8iRCX26dMHn2YT8fS6E4AS320UO2rXI2nTXI7uXknbhDEAxLUfbHa8Lus6ORl/Jy4u7o+3yVYsuwhltIaccZyHUm5zKMV8atex1Nv849VG9O4WwvL1VV8vQL5yilr35fQF8tYLuxBk/rBUXO721Lm8Cl96GduSTG8baBB9/7u1ZYgQldiiRYsKDXEZse36A5B+9ZTNzz2WtAqj0cCIESNk9eVsLZA3jtMaNzKKAewKwhe1nKLWfTl9QdXUC0uoVCrJiwi4vGdsjVohpsePs7KlJ6WJGpUIcDvrOgA+/qFW9zEajRzdsxqthzcDBw50ui9RYxfvRqtV4eWhxt1dTb0IT5590vTaof2H7/+wTRnVoZx34+q6L6cvl1LNesZCNsZqFYwdHkFpqYFfdll+H5cl7hfX17JZAG5uKknPwsupZTDoSdo4B7XajWYPW25kAS6c3EtOxmVadBxmMbpUbl9yajmL/o+G8tIz0eU/X7tZxPQv0jh+UvpjtNWhnGWIUPfl9OVKauwNvMxM03zI4OD7ZzFUlknP1CMuxo95Sy9x6Zr0dDipcX25El5XIKfWz8unceXcIboPeYOQsIZW9zuyawUALTtbf6utqGV0FruTs7l4tRAvTzWN63vTsU2g3e8hrA7lLEOEui+nL5dSkx76MBqNzJ49m8WLF2M0GjEajWg0GsaMGcNLL0mfw2cP40dGMqxfGOu23GDp2qv3P+AuRIxK3PHDTH77dTGtu46kY9+JVvcryMvm1OEthIQ1JKpxO6f7klvLWWRklpCRabosTkrOYef+bGZ/EIuHh5plP1yXpFEdygni1H05fbmS6jabwuafjgULFvDbb7+xcuVK9u/fz4EDB1ixYgWHDh1iwYIFspsZNyKCp4dHsGnbTT6fd97u4zMyiwnw16K18OYAR6ISK6u1c90skjbOoUWnYfR56n2b+6bsX4e+tJiWnR53ui9naFUVaZcKOHshn0G9pAdVVYdyilT35fTlUtQq6YsA2GyM//e///HZZ58RFRVVvi4qKopPP/2UtWvXympk3IgInhkRyY+/pvPp3DSHNESKSty5bha7N8wmrsNQ+o/56L53bI/uXolaoyWuwxCn+nKWVlXirlXjZ8dsCtHLKVrdl9OXK6lREZqlpaUWx4iDg4MpLZUWei2Fp4ebvvTN29P5ZM45h8NDRIlK3LVhNrs3zKZ5+8EMeHrafb/saxeOcfNyKo1bJuDjH+I0X87UkpugAMsjaK2a+VI/yovUM3mStUQup4h1X05fLkWlkr4IgM0xY63WejSerW32MKRPHcaPjOR6ehHJx3T07FKxMcrKLiH5mLRpTCJEJS5ZsoRd62bhHxxO/Yc6kXJgXYXtPn6hNGjWucK6o7tXAtCqs+0hClHKaA054zhfGR9NSJCWwym3uZFRjLtWTZMG3jzaMYiCAj1fLZUeOiNqOUWt+3L6AnnrhV0I0uOVis3GODU1lTZt2pitNxqNFBcXy2IgppEPAGG1PJj8UiOz7YdTdHZ98a6OSjx2zJSCpsu8yoYFfzPbHtXkkQqNcUlxIScObMA/qC4Nm3WtFmW0hpxxnNv2ZPFY12B6dgkh0M8NI6YHPjb8ksHyDddJv2XfXFcRyylq3Zfbl9wxrVKpbjfwqjS1zVI2hSPYej7fXmxlUziq9f+2VVqK8Qmmf+XwBdazKRzVspRB4AjWsikcwVY2hb3YyqawF2vZFI4gZ90Hy9kUjmItm8IR5MimKFg8TfK+XmOmVPrzKouQD30oKCgoVBpBZklIRckzVlBQqLAK3BwAACAASURBVJEUfvex5H09R5kPKVY1VdoztnZppVLBiMQIBvcNJ6y2J9k5xWzblc78JectTlS3NbRQ07QGTjhhtm39vGZm6wAKCg2MeDnV4rb185rJ4qvMm5zf5bBXzpitH9kvmJH9rD/tWao38sTrZyusW/2fxkLXsXuHYyLrejBmaF2a1PcmJMgdjUbFzVvF7D+cw4oN18nMNp+xtGVJW9m0yvQs1TGwv57JXccqTTXrGQsxTPHKs3dyWJetuWTKYR0UQZOGvlZzWB9kLYDjp/L4cUd2hXV6vf0XOXL7kktv75FcrqWb3ySuF+7B0F5BHDwufWqbnL7k1KoV7E5woJbdB7MrZDYP6BFKQscgi5nNVaF1N3LUM7nrmGRq0uPQVYGo+byiapVxPaOEX/dJezt1VfmSU+/C1WIuXDVvjJ8faYpv/GWP9Lv5on6XcmY2y6l1N5WtZ86o+5KpZrMpXP6nQ9R8XlG17sZNA54ejl+Kye3LWeUsw8NdRec2vmRklXDo93yX+HJ2GcGxzGZnalWmnlXF+bKKSi19EQCX94xFzecVVauMzm38SWgfgEajIltXys6DOr5de5P8AulhMHL7cnambqfWvvh4adi4PQeDHZe2on+XcmQ2O0MLKl/PXJolrYwZ24eo+byiagGcPFfArmQd124W4+2lpl2cL4N6BBPX1Ju3ZqRRWCRNR25fzs7U7dnRH4PByC977WtYRP4uQZ7MZmdoyVHPXJolLUiPVyo2G+N58+YxYcIEADZt2kS/fv3Kt33++ee88cYblTYgaj6vqFoAb06vGNqydU8OaVeKGDe0Nok9Q1i+MUOSjty+nJmpG15bS7NGXhw5mc/NTPtuRIn8XYI8mc3O0JKjnrk0S9rJmRP79+9n5syZ/P7772i1Wtq2bcucOXMASEpKYsaMGVy4cIGwsDBeffVV+vfvb1PP5p+OjRs3lv//v//9b4VtO3fudLQMFSgs0qN1s2zDkXzemq5ljdU/ZVBSYuDhFpbfEFIVvpxZzp4dTGlkP9tx484ZvpxRxozMEg6l3CYpOYdFq67xydzzPDsqgicTw+zSkVvLEvbWs6qo+1ZRq6UvdnLgwAFeeOEFnnzySfbs2cOuXbt44YUXALh8+TIvvPACY8eO5cCBA/z9739n8uTJHDlyxLZdWxvvfh7k3mdD5HpWRNR8XlG1rKHXw62cUvz9pN+okduXs8qpVsOjj/ihy9Wz76j9l9vV7bt0JLO5KrTA/nrm0ixptUb6YiefffYZTzzxBImJiXh6euLu7k7Lli0BWLNmDU2bNmXEiBG4u7uTkJBAQkICy5Yts23X1sa7M3jvzeOV642qoubziqplDa2bitBALdk66Zd7cvtyVjkfjvMhyN+NHQdv40hya3X7Lk169mU2V5WWvfXMpVnSdvSMdTodly9fNlt0OvMrsfz8/PJe7rBhw2jfvj0jR45kz549pjKnphIXF1fhmLi4OFJTLT+QVW7X1say1Lb4+HhOnjxJmzZtyn8+dcr2K+elImo+r6ha1n6pxgyphZubiv1HzOeaVoUvZ+iVUZkhCrl9yaklZ2aznFogXz1zaZa0HXnGCxcupGfPnmbLwoULzWR1Oh0Gg4F169bx4YcfsmvXLoYPH87zzz/PpUuXyM3Nxd+/4h8ff39/cnNt/+GxObL/+++/O3AG7EPUfF5RtUYOCCW2oRdHT+aTnlmCp4eadi18aRXrQ+q5fNZtzXSJL2foAQT5a4h/yJtT5wu5eM2x2FZRv0s5M5vl1AL56pkz6oRk7JhNMW7cOIYOHWq2/t5GFcDHxxQxOnz4cJo1Mz0y/sQTT7Bw4UJ27tyJr68vt29X/GOl0+msvvG9DJdPbQNx83lF1Dp2Mo/ocA96dgzAz1eDwQBXbxazcM1N1m6+ZXEKUVWV0Rl6Pdr7o9GoHO4VO8OXXFpyZjbLnf8sZz2Tu05Ixo6hVH9/f4sNryX8/PwqvIruzseZPi82NtZsgkNKSgqxsbE2dYVojA0GWLb2MsvW2vfX+0HU2nckl31H5Btnk7OMztBbtSWLVVuyKq0j4ne5Y18WO/ZVvmxya4G89UzuOiEVoxMfhx49ejRff/01/fv3p3Hjxqxdu5YrV67QrVs3jEYj8+fPZ9WqVSQmJpKUlMS2bdssDnncjRKhqaCgUCMp2LZE8r5eCaPt0jYajXzxxRcsW7aM/Px8mjRpwptvvskjjzwCmOYZT58+vXye8WuvvXbfecZKY/yAkJaWxg8//MDu3bu5ePEiRUVFREdH07dvX8aNG4e3t7fZMb/++isLFiwgJSWF4uJiwsLC6Ny5M++8844LSqCgYB8Fv34neV+vR0c50Yk0hMgzfnXCnYi9vcmZpoi9geEcScmxGLFnK2tW0bIcSTik+wEWLFpC4xY9iO81CLXGjUun9jFz5kyWrNjEU39Zjtbds3z/pA2zSdo4i/oPdaFd75dxc/dCl3WVfcdOWvVVmXLm5hu4eKOUemFuFBQZmfxltpl2WIiaDyYGcfpSCSnnSsjJNRAaqObRNp64a1Vo3VRKHbOzXljLphZBq7IYBXnrs1RcPmYsarxhTdPq06cPnjET8fC6E8zSuusogmrXY++PczmWtJI2j44B4EJqEkkbZ9F54Ct07DdJkq/KeJv8ZRYZ2aansN6bEICH1vIv0e18I/+cn82lmxXnuO5LKeYffwqweExlfClaYmnZTTXLpnC5W1HjDWuaVosWLSo0xGXEtDGNY926dmfe+N6f5uLtF0L73hMBKC7Mw2i4/yOrjnora4jvR16B0awhBriWoedquvWHEEQ4/4qWKyI0pc8zFgGX94xFjTd8ELQAbmdfB8DbL9SkU5TP5TMHadi8G8eSVrJn0xfk5tzETetJo5Y96PH420BolXiTigoI8LXerxD1/Ctazo3QdOZsCmfg8p7x/SL2ggLccbPwXLuiVXktg0HP3k1zUKvdeOjhgQBkp1/EaNBzNe0IW1d+RIvOTzB4wmxadX2SU7/9yPf/fpqCggKne7OH7m08CPSzXpVFPf+KlvPqBFCzwuWvXr1KeHi4Uw2IGm/4IGhtWzmNq2mH6Jr4BsF1Gpo0Ck1zSwtyM+n91Ie07DwCgCatH8Pd05c9G2ezZs0aIMJMzxVxiY0i3Hiilw+XbpQSVcdydRb1/Ctazo3QNArSyErFpttJk+7cvHn55ZedYkDUeMOarrVr3UwObV9My84jad9nYvl6tz9mVKhUapo9MrjCMc3bDwFMOa7O9CaV6DANLz/hR/ZtA//53npWgojnX9GqggjNajZmLDlC89KlS04xIGq8YU3W2r1hFnt/nENch2E8Nur9Ctv8Ak25t57e/rhp3Sts8w0w3WyxlGQllzepRNfR8MYofwqKjPzfEh3ZudZ/oUU7/4qWY1r2YlSpJS8i4HCEplyIGm9YU7V2b5jFno2zad5+KH1Gf2T2vfr4h+IXFE5Bfg4lxRXHhm9nmW72BQcHO8WbVKLraHjjKX8Ki418tkRHps52z0qk869oOa5lN07MM3YGDkdotmnTRhYDosYb1kStpI2z2bNxNs0eGUzfMdNQWXnDQfP2iWA0cmTX9xXWH95peqKpe3fLE/KrIi4xqo6G10eZGuL/W6IjI+f+l7iinH9Fq3Ja9mJUqSQvIqBEaD4gWkuWLCFpwyz8gsKpF9uJ3w+uq7Dd2y+U+g91BuDhXhM4dWgz21d/TNaNNGpFxnLlbDK/H1hHdNMO9O/fn4//u0s2bx3i3AkJMPVO/LzVaNQwoLMXALdy9Ow9borODPZX8/oof7y9VGw9WEijCDca3XMf0dNDbTYGKcL5V7TEjtAUAZfPMwYx4w1rmtaxY8cAuJ11lU2L/ma2PbLJI+WNsYeXL0++sYTd6/7NmWO/cGzPKvwC69C+z/N07PciGhvzNx3x1qWVJzH1tBXWDeluyso4eaGkvDEODVTj5236BUvsZp6lARAYoOX6zSJZfMlZRkWr6iM0jYjR45VKlQYFWcsNsBdbz+crWtb15v0sixQTesnra8K0W7JozZsSotQxB/RE1aos2Ye2St43ML5HpT+vsgjRM1ZQUFCQnWo2TKFEaCooKNRIMo/uvP9OfxDcsqsTnUhDiAhNlQpGJEYwuG84YbU9yc4pZtuudOYvOW9xQritSzURtBKe2FdhfWRdTx7rGkK7loGEh3ngrlVz9UYhv+7JZNXG6xa1ti1vL5uvMm9yaX37n2je/r8fuHRqN7pbF9GXFuEfHE2jVn1p2fVptO53xnMPb/+GCye2kZ1+nsL8bDy9Awms3YAWncfSsMVjvJZoPfbSkfMvch0bOOGE2bb185pZ9FtQaGDEy+ZvEy7bX8R6IbdWpRFkloRUhBimeOXZO3mny9ZcMuWdDoqgSUNfq3mn1UmrX0IthvSpQ9LBLH7elUGp3kh8c3+eHRVFQsdgXnw7heIS6cZcXcZVq1ZxdOcS6jfrQdP4gag1Wq6c3cf+H2dy9sgmhr38PW5a05N8Ny8dwy8ogujY7nj6BFGUn83Zoz/x06KXebj3K5BoPaLT1eWsCi2A46fy+HFHxfxmvd7+C1ZRyyj3+ZKKKA9zSMXljbGo2alyam3fm8nSNVfJK7jz/P26LTe5fK2QscMj6N+jNmt/kjbfUoQy9unTB33UcxUiOZt3fJJ9ofX47Ze5/L5/JS06m7KRe4/5l9nxLbuOY+W/h3N4+3z0+ueFLaeztcq4nlHCr/ty7DrGmb5E1bKX6jabwuV/OkTNTpVT69S5vAoNcRnbkkyzCBpEe7nEl9zZyI1b9QMg8/ppm5+r1rjhE1CHkuICSktLZfVWnbTuxk0Dnh6ONx6iltGVecbV7XFol/eMRc1OrYoc1lohpuyHrGzpr1EXuYy5OabevZdviNm2wvxsjAYDhXlZnD36IxdP7iSiUXs8PDyc7k1UrTI6t/EnoX0AGo2KbF0pOw/q+HbtTfILpAfoiFpGV+YZ16gx459//pkbN24werTpzakjRowgMzMTgLfeeou+fftW2sD98k5bNgvAzU0lKUxEVC1LqFUwdngEpaUGftklfZ6tqGU0GPQk//wlarUbTeIHmm3/7uO+FOabxkXVajcatuhNt6HvVok3UbUATp4rYFeyjms3i/H2UtMuzpdBPYKJa+rNWzPSKCySpiNqGZ39e2QLg0qMzAmp2GyM58+fz7/+dWfMr7i4mJUrV1JQUMDkyZNlaYxFzU51dg7rpGfqERfjx7yll7h0rVDycaKWcfcP07hx4TDt+71OUO2GZtv7jJuFvqSIPN0Nzh79CX1JISVFeYB5L1pub6JqAbw5Pa3Cz1v35JB2pYhxQ2uT2DOE5RszJOmIWkYlz1g6Nt2WlJRQt27d8p/btm1LUFAQ4eHhVt/2YC+iZqc6M4d1/MhIhvULY92WGyxde9WuY0Us4/4f/83x3Uto1v4J2vSYaHGf8IYPExXThdiHhzPgz/9F6+HDmi+eIifH8o0rEcspt5Y1Vv+UQUmJgYdb+Eo+RtQyujLP2IhK8iICNhvje3Nr33nnnfL/lw1XVBZRs1OdlcM6bkQETw+PYNO2m3w+77zdx4tWxgObZ5H8yxxiHx5Gt+Hv29z3bmLaDiX/djqbN292mjfRtayh18OtnFL8/aRfZotaRiXPWDo2XbRs2ZLly5ebrV+2bBktW7aUxYCo2anOyGEdNyKCZ0ZE8uOv6Xw6N+3+BzjZV2W1DmyexcEtXxDTdgiPPv6hXZnXpSWmoRlrPWORyuksLWto3VSEBmrJ1km/dBe1jK7MM65uEZo2G+MpU6awevVqxo4dy4wZM5gxYwZjx45lzZo1TJkyRRYDomanyp3D+vRwU0O8eXs6n8w55/BEd1HKeHDLFxzc8gVN2wwm4QnL2cglxfl/jAtXxGDQczxpKQCtW7eW3Vt10fLzsdzzHTOkFm5uKvYfsf4qKWf6ElXLXgwqjeRFBGzewAsJCWHZsmXs2bOHM2fOAKZg8Y4dO8pmQNTsVDm1hvSpw/iRkVxPLyL5mI6eXSretMrKLiH5mOVXGYlYxiVLlnBg8yx8A8OJbNKR04fWV9ju5RdCVNPO5KRf4H9zx9KwRR8CazXAwzuAvJwbnDm8gez0NGLaDqFdu3aA+SO0IpTT2VojB4QS29CLoyfzSc8swdNDTbsWvrSK9SH1XD7rtkofChS1jK7MMxZlLFgqkuYZd+zYUdYG+F5EzU6VSyumkQ8AYbU8mPxSI7Pth1N0khtjOX05qlWWjZybfZWt3//dbHt4w4eJatoZn8AwmrZJ5FpaMmnHf6akKA93T19CI5rRtteLFqfAiVROZ2sdO5lHdLgHPTsG4OerwWCAqzeLWbjmJms337I4Hay6lVFuLXsQZSxYKkqesRO07g0KcgRbQUGOIHfW7Mwf5Kk2toKC7MUZebpy1gtLQUH2YisoyBFqcp7x+TOnJO9bv3HTSn9eZVEiNBUUFGokaWfPSN63QaPGTnQiDZc/Dq2goKDgDGrkmLFcdBtq/hJLgB1rupBzu4RTZ3OJaeRLXoGekRMPWtXZsaYLYPlS7dUJd+L69iZnmuL6BoZzJCXHYlyfrcvRmqJVpudsLUfLmX/+Mp7htbk473tup5zGr3kToieM5PaJMyR1G4WhwDQNzrthFJ2TVmAs1ZP2n4WU6nKJ+vMI/Jo3Yf/ACXT4aYGsvkDe79JS/Xdl3S/zdu+w2oSnosojX0+czi2PfE3oFMLZ83lWI1+3LW8vq6/KYnB9DppdCNEzHvn8Aa7dML1EcsG/4/HydGyqiajRfw+CVmX0vOtHcmjMG1z9fkP5uqw9h4hf/DkNXxvPmelzAIj58E20gf7saj8M3RFT8Prlb9fS7ch64v5jPedC5HMmYt0XNfLVXozVrDEWwm1ZZawsokb/PQhaldHT5xdUaIgBri7fiL6gkMhxwwDQeHtRZ1APbm3fX94QA+jz8rn0zUp8YxpUSTnlPmci1n1RI1/tpbo9Di1Ez1guRI3+exC0KqOnL7TQIBmN6AsK8WkUjTYkCJ8m9dB4epC197DZrtn7zNfJ4cvZWnLyIES+2osojaxUhOgZy8X94vqCAtxxs/CMvKJVea3K6LkHB+LfKrbCOv9WsbgHBwLgFV0Xz7qmHlThVfPLY0vr5PDlbC05cbYvZ0W+OvN8VbeecY1qjKXG9Sla8mtVRs+o1xO/dCa1+nbDM6outfp0I37JTAzFxYBpiELjbbo0NhQVmx1vsWctgy9na8mJs32VRb7+v+VXnBL56gyqW2Nsc5jigw8+sBn+MnXqVNkNVYbCIj1Bnu4WtzkS/ado2Rdv6Kjeb6PfoPm/3uaRdfMAMJSWcumbleSeCCZsaG9Kdbno802RrWoPc32Np+W3hVTWl7O15MSZviob+eqq82UwVq++pk23cXFxNG/enObNm7N169by/5ctoiFq9N+DoFUZveurfuSX+t3Z2W4wSY8+xS/RXTk+6V08I8MwlJSQd+YChddMd9w9w+uYHW9pnRy+nK0lJw9C5Ku9VEXPeNKkScTExLBv353pgUlJSSQmJtKqVSv69OnDxo0bJWnZbIyHDh1avvj7+1f4eejQoQ4XwFmIGv33IGhVWs9gQHcklazdyRSnZ+JRJxT/1g+RueMAhoJCbh87hb6wiKAO5ilvge0tJ7/J4suJWnLyIES+2ouzG+O1a9dSWFhxyOby5cu88MILjB07lgMHDvD3v/+dyZMnc+TIkfvqSe7H25NV6ypEjf57ELRk1VOpaPavqag0Gs5MnwuYprDdXL+NkO6P4NcypnxXjY83UX96nNxT1hsLkc+ZXDwIka/2YjSqJC/2cv36dWbOnMkHH3xQYf2aNWto2rQpI0aMwN3dnYSEBBISEli2bNl9NYWY2ta7ey3CankCEOivReum5unHowC4nl7I5u3pknREjf57ELQqo9ftyAZu/G8L+ecv4+bvR/iTAwlsG0fq1M+5tf3O5V/q1M8I6dGB9hu/Ie3fCyi9nUfUn0fgGVGHA4kTab/x62p3zkSs+6JGvtqLwY4er06nM3uzEYC/vz/+/hV79UajkSlTpvDCCy8QHh5eYVtqaipxcXEV1sXFxbFhQ8V59Jaw2RjHx8eX94gLCwtp06ZNuRmVSsVvv/123w+QwoBeYcTHBVRY9+zoegAcOp4juUKCuNF/D4KWo3q6o6mEPzkIj7q10OcXkHPwGPv6/5mMLRUfH84/e5E93UcR89FfaPTX51C7a8k5dIL9A57l1tY9VVZOObVErPsiR77agz038BYuXMjs2bPN1r/00ku8/PLLFdYtXboUo9HIyJEjzfbPzc2lceOKoUP+/v7k5t5/OMZmY3zo0KH7CsjBq/84JpuWwQDL1l5m2drLilYVazmqd3jsm5L3zU09R/LwF6vEV1VoiVj3P/7yHB9/eU4mV/LXManYMxY8btw4i/fB7u0VX7x4kTlz5vD9999b1PH19eX27YpvaNHpdPj63v/lskIMUygoKCjIjT1jwZaGIyxx8OBBsrOzGTZsWIX1L774IgMHDiQ2NpadO3dW2JaSkkJsbMWHmiyh5BkrKCjUSA6czJa878MxgZL2KygoMHuJbvfu3Zk5cyadOnVCp9MxcOBA3nnnHRITE0lKSuLVV19l4cKFtGrVyqZ2lfaMrcUbqlQwIjGCwX3DCavtSXZOMdt2pTN/yXmLE8JtxRsqWpYn0O9a150hL1p+84Gvt5rH+wbTvqUvIUFuFBQauHitmO/W3eLE2QKz/dd+2VTW73JXioEGYSqCfECjgexcSL1s5MApIyV35dXUDYa4eirCglTUDjBNj1q/38Cx86b+xOQnNELXMUsRmqOHRdK0kS8xDX0JD/Pk2s1ChyM0Ha0XompVFkdmSdwPLy8vvLzMg5KCg4MJCAggICCAOXPmMH36dN5//33CwsKYNm3afRtiEGSY4pVn7+SdLltzyZR3OiiCJg19readKlryaNUKduPD16Lw9FDxc5KOqzeL8fZSUz/Cg+BA+6uHI94ebqrizFUjKRfAYIToWtC9hZrYKCOLfjFQ+keD3KiuijaNVNy6DTdzIDLUub6qQmvi2Prleca+Vt4W7QpfomrZQ1U9B3ny5MkKP3fq1Il169bZrePyxljUrNkHQQvg9WfC0GjgtY8ukKUzj020B0e9fbHeQNFdQWCHzkJWroHOzdS0aqAi+Yzpt/W3M0b2pZp6yzGREBkqrfES+fyLmGcsqpa91KjHoasCUbNmHwStZo29aNbYmzVbssjS6dGoTZf+juKotyILiYy/XzI1wKF3zfrKL6LCsIWzfTlbC8TMMxZVy16c+dCHM3B5z1jUrNkHQattc9N80vTMEt5+IZw2zXzQaFRcuVHM8k232L7/9n0UnOfN749huXzpAWFV4utByDMWVcteREljk4rLe8aiZs0+CFoRdUxpWi+OroOvt4Z/L7rOfxZdp1Rv5PVn6tKjw/2n+jjDm0oFnZup0RuMpFys/ICiqOdfTkQtoyvPl8EofREBlzfGombNPghaXp5lT1ca+MfMS+w4cJute3VM+ewSufl6xgwOxZ5IErm89WqtIjJUxc7jRjLt65w71ZfcWnIiahmVPGPpSH4c+m7kfBxa1KzZB0Gr7A2/Ow7eLp+xAJBXYGD/0Vx6dAggoo47l6+bB7o7y1u3OBXtmqg5dNbAnlR5uiyinn85EbWMrjxfoowFS8Xlj0NnZBZTP8oHrZvK7FLGkaxZRcsOraxSALItzKLIyjGt8/GW3muprLcuzVV0bqbmSJqBH5Plu3YU9fzLiahldOX50lezxtjlwxSiZs0+CFqnz5vujoVYmE8cEmRal3Nb+vSFynjr0lxF1+ZqjqYZ2HhA3l9OUc+/nIhaRpfmGVez2RQub4xFzZp9ELT2Hcklv0BP90f88PS4UyGD/DW0b+nLlRvFXE+X/iZgR711bmZqiI+dN7BB5oa4Mr6crSUnopbRtXnG0hcRcPnUNlGzZh8ErbwCAwtWZ/Di6Dp8/FY0vyTl4Oamom/XQNzcVMxbXkV5xnFqcvKMnL8BzaMr9lLyikzrAfy9TY9DA9T6Y/5x43BV+TQ4uX05WwvEzDMWVcteRLkxJxWXN8Ygbtbsg6C1eXcOujw9Qx8L4qlBoRiNcDKtgM//3zVSz9k/yddRbwE+Kga1N//luXDTyPkbphs8gT6mx6TvJjZSRWzk/X/pRD3/IuYZi6xlD6JMWZOKEI2xqFmzD4IWwN7Duew9LM/YnSPepi+XNi59Md32vpOfsP4osajnX8Q8Y5G17Pvc6tUzViI0FRQUaiTrfyuVvO/ANq7vl7regYKCgoITqG7dTCHyjF+dcCdib29ypilib2A4R1JyLEbs2cqaVbQsRxJay5qVU8vRclrK+d2xpkt5tGRMI1/yCvQ2c37LjpG7juUXGrmaaSQ8REVRCXy20vLsknZN1Azp7MaVDANHzhkoLoXIUBXxjdVo1KbL5dwCA5du6Imuo6Gw2Mjbcy2/R+6NUb5cv2XgZpae/EIj/j5q2jd3p+5dKXXVqV44qlVZRJmyJhWX94xFjetTtOyPN5RTT65oycr6mrbM1Pi+PNjNZqJd5zgNunwj8zaVlj/NePAU5BbAo61M3v/yH9MbIv7xJ3+sPJQGwOffmY/fb0su5IOJAfj7WJ6NKmq9cG2EplNknYbL5xmLGtenaNkfbyinnlzRknL7soanFgqKjBUeKwfQ5cvTIhSVQF6BdS1R64VrIzSVecZ2IWpcn6Jlf7zhgxAvaY3TVw20aayhbzsNyaf1FJdARKiKhFaO9+h9vFSoVeDvo6Zrq4rDFPciar1wZZ2obo9DW22MrYUEAbi7uxMdHc3rr79Ox44dK2XgfhF7LZsF4OamkvT8uqLlOi1n6MlFVfjasE+PVgMdm6npEnen0Uw+radtE/sbZA8tfPbKnZdkFpcY2XG4iG6tPSzutUIb0gAAHYdJREFUL2q9cGWdEKXHKxWrjbGtkCC9Xs/p06f5y1/+wvr16ytlQGrEXu6913+KllBaztCTi6rwZTBATh78ftFI6iU9JaVGmkSoadPYsZHA4lKYuew2ajWE+Kt5pLk7nlrr+4taL1xZJ2pMY2wLjUZDbGwsY8aMqbQBUeP6FC374w0fhHhJS6iAcY+5oVbDfzfemduackFPfhF0a2F/z9hohNQLd7R2HS3mjVG+VvcXtV64sk4YqtkwRaVu4D355JOVNpCRWUyAvxathbR/R+L6FC3XaDlDTy6c7ateHRX1w9SkXDBvVI6fl6ehMRph/wnrudKi1gtX1onqdgPP5bMpRI3rU7Tsjzd8EOIlLeHvbWpoLN1iUcvYOXO38XoiUeuFK+uE3iB9EQGXN8aixvUpWvbHGz4I8ZKWuJlt6lq1aqg2a3zj7Rwz9vZQobFwiLsWOrW0fPMOxK0Xro3QrF55xi6f2iZqXJ+iZX+8oZx6ckVLVtbXoy1NLaOPp6mRLPs5OxcOnzN1qa5nGTl+3kBcfTUvDHLjyFkDJXpoHK7moeg7LWu/jqby+Hmr0Gju/JypM7AvxTQE0STajdF9vDl0spib2QaKio2EBKhp39yDYH/rDbuo9cKlEZqCDD9IxeWNMYgb16do2aclp56c0ZKV8dXrngCZsp/TrhvKG2OAFTtKuZyhplVDNT3jNahUpgZ7+1E93VuabuAN7lYxeLns51MXS8ob4yvpeo6eKaFptJZHmqtxd4PcAiMXrpey9KciXhphfV6uqPVCidCUhhCNsahxfYqW6/TkjJYEx31NXSDtZax6A+w6bmDXcfMByLLG+PmPs+6rk5FtYPGP+XZ5LEPUeuGqCE2lZ6ygoKAgANWtMVbyjBUUFGok836Wvu+EXs7zIZUq7RlbikoEGD0skqaNfIlp6Et4mCfXbhbajEvcsaYLYDlGUKWCEYkRDO4bTlhtT7Jzitm2K535S85bnFxuK6qypmiV6SU8sc9sfWRdTx7rGkK7loGEh3ngrlVz9UYhv+7JZNXG6xb1ti1vbzWq0pFyDpxwwoISrJ/XzOL6gkIDI15Otbi/nL5A3u/yxrfTmf/rIbadSOOGLg8fdy2N6gQzqdfDtGlQF4CZP+7lt/PXuHgrh9zCYoJ9vWgaFsK4rq14uGEEnsNfl81XmTdRtSqLQZApa1IRYphi4tj65dm1vj6OB6sAvPLsnezUZWsumbJTB0XQpKGv1ezUB1mrX0IthvSpQ9LBLH7elUGp3kh8c3+eHRVFQsdgXnw7heIS6YJyegM4fiqPH3dkV1in19t/Mefq83/lyhXGfLGKgqIShrSLpV5oILmFRZy6nskNXV75fscu3aBJWAg9mzfE38uDW7n5bDh0mmfnr+PDET0YMVzcMlaFlj1Ut2t+IRpjubJrRc1hFVULYPveTJauuUpewZ1sgHVbbnL5WiFjh0fQv0dt1v4kbS6oM7Jrr2eU8Ou+HLuOcaYvR7Xeeust9AYDK14ZQS1/H6v6X08YbLbuqY4tGPB/S/nm10OMELiMztayl+rWGLv8oQ+QL7tW1BxWUbUATp3Lq9AQl7Et6RYADaK9zLZVlbcy3DTg6eH4xHxXn/8DBw6QnJzMM11bU8vfhxK9noJiy28MsYS3h5ZAb090hdZ/T1xdxqrQsheDUfoiAkL0jOVC1BxWUbVsUSvEFO6SlS290XCGt85t/EloH4BGoyJbV8rOgzq+XXuT/ALpA4KuPv/bt5vGUesG+vLyok3sPnURvcFIdEgAE3u0ZWB8U7NjsvIKMBgh43Yeqw78zrn0LIa0jRW2jFWhZS/2zU1w/VN4Nhvja9euUbduXYvbtm3bRkJCglNMOYqoOayiallDrYKxwyMoLTXwy65bko+T29vJcwXsStZx7WYx3l5q2sX5MqhHMHFNvXlrRhqFRVWfs+yIVlpaGgDvr9lOvZAAPng8gRK9gUU7j/D2iq2UGgwVGtr8ohIe/Whh+c+eWjeGP/wQfxnQSdgyVoWWveirNqm10thsjMePH8/8+fOJjIyssH7lypXMnTtXuMZY1BxWUbWsMemZesTF+DFv6SUuXSuUfJzc3t6cnlbh5617cki7UsS4obVJ7BnC8o0ZVe7LEa28PNMNOh8Pd+Y/m4jWzXRPpEezBvT/dAmzNu8nMT4G9R/BFh5aDV/9aSB6g4GrWblsPHKaguISCoutv3re1WWsCi17qVFjxpMnT+bPf/4z58+fL1/31VdfsXDhQhYvXuxsb3ZTWKRH62a5SI7ksNZ0LUuMHxnJsH5hrNtyg6Vrr9p1rLO9Aaz+KYOSEgMPt7Ce7etMX45oeXqaMij6tWxc3hAD+Ht58OhD9cm4nc/5jDszRjRqNR0aR9K5aTQj2jdj/rODuJady4Sv11FSYnnYyNVlrAote6luY8Y2G+Pu3bvz3nvvMWHCBE6dOsVHH33Etm3bWLx4MWFhYVXlUTKi5rCKqnUv40ZE8PTwCDZtu8nn887bfXxVZNfq9XArpxR/P+kzblx9/uvUqQNAiJ+3+TF/rNMVWL85p1Gr6d+6CWduZHLwoOX5964uY1Vo2UuNyzPu2LEj06dPZ+zYsVy6dImFCxcSEBBwv8Ncgqg5rKJq3c24ERE8MyKSH39N59O5afc/oAq93Y3WTUVooJZsnfTLWlef/5YtWwJwI8d8W9kc42Bf27NWikpMQxTZ2dkWt7u6jFWhZS9Gg1HyIgI2G+P4+HjatGnDhAkTyMvLY+/evXTs2LF8vWiImsMqqlYZTw83NcSbt6fzyZxzDvcU5PTmZ+XhnzFDauHmpmL/kdsu8eWIVq9evfDx8WHD4dPkF90ZZkjX5bHtRBr1QgOIDglAV1BEiYWx0/ziEtYcTEWtUpU37KKVsSq07MVZ4fKffvopAwYMoE2bNnTp0oUpU6aQlVUxBOrEiRM8+eSTtGrVikcffZRFixbdV9fmDTxbLyWVE7mya0XNYRVVC2BInzqMHxnJ9fQiko/p6NklpML2rOwSko/prBztPG8jB4QS29CLoyfzSc8swdNDTbsWvrSK9SH1XD7rtmZK1nL1+Q8ICOBvf/sb77zzDmPmrGZIu1hK9AZW7EuhRG/g74NMj/cfTLvKB2t20CuuAVEhAfh4aLmSeZv1h09xIyeP53u0JSIiwkxfhDJWhZa9GJzU49VoNHz66ac0adIEnU7HW2+9xeTJk5k7dy4Aubm5PPvsszz11FMsXLiQ33//neeee47atWvTt29fq7pCzDOWM7tW1BxWUbViGpmeBgur5cHklxqZbT+copPcGMvp7djJPKLDPejZMQA/Xw0GA1y9WczCNTdZu/mWxalSVeHLUa2RI0fik7qLBTsP88WWA6ZebnQdpo/sSXw90/TRJnWC6f5QPQ6eu8bGw2coLCklwNuD5pG1mTq4G91i6wldxqrQsgd7tHU6HTqdeT339/fH37/iEMsbb7xR/v+QkBDGjh3Lm2++Wb5u8+bNqNVqXnzxRdRqNa1bt2bEiBEsXbpU/MZYzuxaUXNYRdX6+MtzfPzluUrrlCGXt31Hctl3RL7xRBHOf6+4hvSKa2h1e1RIAO8Ne7TKfVUnLXuwpzFeuHAhs2fPNlv/0ksv8fLLL9s8ds+ePcTG3pknnpqaSrNmzVCr74wCx8XFsWLFCps6SoSmgoJCjeSD76zPy76XVwfkS+4Z383GjRt5++23Wbx4Mc2bNwdgypQpFBUV8dlnn5Xvt3v3biZMmMCJE5YTCkGQnrGCgoKC3BjtuDF3v0bXEhs2bOC9995jzpw55Q0xgK+vL7duVXxyVafT4etre258lTbG1rJmX51wJ2Jvb3KmKWJvYDhHUnIsRuzZypqtaVr9njlq8ZzdzbMj69Lv0WC8vTT88HMGcxabP6yxaUFLWXyVeZPzuxRVC+T9Li3lee9Y06U8PjamkS95BXpJWd55hUauZhiICFVTVAIfLzOfp6x1g+b1LU+YGtxJi9YN3DQqp9cLR7UqiyNRq1JZsWIFn376KXPnzqVt27YVtsXGxrJp0yYMBkP5UEVKSkqFoQxLuLxnLGpcn6ha99KonhdDeofy9fJrPDcq3K5j5fYl6jkTVQscj4/94FvTMa8Nd8dDaznkpqQUDp8x7x5G11bh5aHi6Dk9LRuaf57I58senDUCu2jRIr744gu+/vprWrRoYba9d+/e/N///R9z5sxhwoQJpKamsmLFCt577z2bui6P0BQ1rk9UrbtRq+DV8REcPHab3cn2Z/7K7UvUcyaqFsgXH2sPD8eaGuADJy0/OCPy+bIHZz0O/dFHH5Gbm8vTTz9NfHx8+XL1qumK1NfXl/nz57Njxw7atWvHyy+/zKRJk+jXr59NXZf3jEWN6xNV626G9gklqq4nH84+6dDxcvsS9ZyJquUK3N2gZQMNWbeNnLlseVC1ppwvZz1Zd/Lk/X/fmjVrxvfff2+Xrst7xveL2AsKcMfNwnPtD6pWGXVCtYwZGsbS/93gZob0zGFn+hL1nImq5QpaNtLg4a7i4KlSrDVVNeV81bhsCmssWLBAFgNSI/YUrYq8PO7/t3emUVFd6Rp+igJkFkRFxYAxKiDBARVCaIcQDU7BmNWi0Ri7s7RjEu20WZqeVvftpKNXYvcNNth4O9qmVbzGiCSQOMe0aSbnAZOoqDiPiQyFzHXq/qiIKSmKoupAbXQ/a+0fdU6dt779cerj1D57v6cn127WsGWH9QtiWjsuUXMmqpYjGBaiRVEMHGxiiAIenHwpisHqJgIOL8ai2vWJqgXwVIwvg8O9WLH2il0G2mrHJWrORNVqa7r6aggOcKLoikLZnabf96DkS9EbrG4iYHMxVutOpah2faJquThr+MULPThwXMftsnq6d3Wle1dXAn54TJKnh5buXV3x9Gj+T6u2vaGoORNVq60ZFmL5xt1dHpR8KQaD1U0EbC7GGo064zyi2vWJquXqqsHXx5noQT78873Qhvbeb42+Ek8/6cc/3wtl7IhObRqX2noPg1ZbonWCwX21VFQZ+Oa85SvRByVfBoPB6iYCVllo3t8GDx7MzZvqzA0U1a5PVK3qGoXFqRcatdR/Gdf9HzhezuLUCxQcad7cR217Q1FzJqpWWxIW7ISXu4YjRfpmp3I9KPlqb2PGDrfQFNWuT1QtvR5yDjaeU9y1swsA127Wmt3f2nGprfcwaIHt9rFxg41DDp5uGpy1916X6AwcMbPQ4+4Qxf5mhijU7qMjLTQFueC1GofPMwZx7fpE1VITteMSNWeiatlqH/vMUBezr89dVThyptZkX0dP6BvoxPnrCrdKrQtQ1Hy1BH1LXeMdjBDFWFS7PlG1zHHzuzqrfCzuR+24RM2ZqFq22sf+5gPrn9pddgd+t7plK/1EzVdLEOVxStYiRDGWSCQStWlvxVj6GUskkgeSV5eZf3irOdIW+bZiJNYhhIWmRgNTEgKZNLYH3bq6UVpWy5c5t1iVft7shHBL9oYiaJmzSpzxfE/6PeZFSG8venRz49rNaqusEtWI625sra1li15O9khV8gXGnIl8jh07W0/PLk74+zhxu1xhSbrloYbIflpi+jvT3d8JjcZ4c65bJyfV4gJY97cg/rQ8iwvf5lL63UXq62rw7RxESORYIp+ahWsHj0bHnD3xbw7t+ZAbF79GX1+Lt183gkNj+eiDP6p6jtlLe7syFmKY4pez7/mdbsy8ZPQ7fTaQvr29mvQ7bW9ar8zs1eBb69XEk48dEZeaWmrqqZkvNeOyR6tPoJYrtxTcXZv/sMRRrgwN0VJ4Ts/hojoMBujkrWkoxmrFlZGRwaE96fSJiCNs2LM4aZ25dHofOdnJnDq8jekLN+Hi6tbw/rzPU8nbmkKvsJ8QO2E+zq7ulJdc5bsrTZvnqH2OWUt7+9Hv8GIsqneqKL61rRmXyH7GauVL7bjs0fqvNVUALJzqhquFb15UqJaoMGc2fFHD4dOmU9HiIl3MHmNrXPHx8biFvEIH93vuaYOGv4Bf12AKtq+kMG8zkaNeBODCyTzytqYQO/GXxIx7vekOqBCXGrS32RQOdzQR1TtVVN9akfuopp6aPr8i58wccZEuXL6lNBTiDubrrypxRUREmBTiu4REjgfg+2unG7YV7FiJh7c/0c+8AkBt9R0MiuWC50g/Y4NisLqJgMOvjEX1ThXVt1bkPsqc2d/HLr4aOnd0IqewjtFDnBk+wAVPNw1VNQaOnmn6AZtqx6UrvQ6Ah3dno05NJZfPHKR3+AgK8zaTv20FFWU3cXZx47EBccT99PdA51aPqyWIUmStxeFXxqJ6p4rqWytyH2XO7O9jV1/jV3JQH2dGDnRhz+E6/rWjhhPFemLCm75EVjMuRdFTsC0NJydnwoZNBKD01kUMip6rxcfYs3kxEbGJTJqTysDh0zh9eDsfLX+JqqqqVo2rpbQ3oyCLV8Zz5861ePDKlSvtDsBav9OK+uaXcYqqpSYi91HmzP4+3h2S8HLX8L9Z1RRdMWoWnjPqDQs1/5VVM64vNy/havERhie8SaeA3kaNaqOhT1XFbZ6Z/i4DYqcA0HfQGFzdvMjfmkpmZiZg6kHhyHOivV0ZWyzGR48epXv37kyYMIGBAwe2yt3J6ho9fm6uZvfZ4p0qopaaiNxHmTP7+1j3Q00qrVAaCvFdDp6qb7IYqxVXTnYyR/auZ0DsVKLjX2nY7vzDjAqNxon+UZNMjgmPfo78rans378fmNwqcdlCe5tNYXGYIjc3lwULFlBUVMTixYvJzc3Fz8+PqKgooqKiVAlAVO9UUX1rRe6jzJn9fSyrMB6rq2ysUW5mm5px5X6eQsH2NB5/4nnGvPC2yT5v324AuHn44OxiWly9OhpvwpWXN3YKdOQ5oa9XrG4iYLEYa7VaRowYQVJSEps2bSI4OJiZM2eyfv161QIQ1TtVVN9akfsoc2Z/H6/dVqirN9DRs/FX09ez6bFVe+PK/TyF/K2phEdPJn7G4kZ+5Z4+nfH260FVZRl1taZjw7oS482+Tp0ae2hLP2PrafYGXm1tLTt37mThwoWkp6czc+ZMxowZo1oAonqniupbK3IfZc7s72NdPRw/p8fHU8Pjj5rOrY55vOlRRXviytuaSv7WVPpHTWLsi0vQOJkvC+HRCWAwcCzH9KnHR//zfwCMHNl41ZwjzwmDoljdRMDimPFbb71FUVERI0aMYN68efTr10/1AET1ThXFt7Y99VFNPbXypXZc9mg9HWn8unm6adBq770uqTCYLO7Ytq+Ovj21zBjtSk5hPSU6A6FBWvr3anrhi61xpaenk/d5Ct5+PQgOfZJvD2ab7Pfw7kyvsFgAho2ew+kjO9m7JYmSG8V06RnKlbOH+PZANkH9nmD8+PEk/cN0absj/YxFMY23FovFOCsrC3d3d86fP8+6desathsMBjQaDYcPH1YlCFG9U0XwrW3tuET1M1YzX2rGZY/WuGhXs6/PXtGbFOPSCgMpW6oZF+XCsFBn3Fzh+3ID2Xm1PPuk+ZthtsZVWGi08NSVXGXb2l832t+zb1RDMe7g7sW0N9PJzV7OmcIvKMzPwNs3gOj4ucSMew2t1vw/C0f5GYsy/GAtbera1pSJS0uxZOIigpY545uWYskoyBYsmfs4WkuNfIFlo6CW0hrnxcK0Sru1/vKq0bhHzX5+sFsVKeaMVv87bg8/feOc1e/dvLy33Z9nLw5fgSeRSCStgWIQYyzYWqSfsUQiUYXi4mKysrLIzc3l4sWL1NTUEBQUxNixY5k1axYeHo3tOO+yYcMG3n7bOJ0uPz/f7MyMljJ5XpHV781M7Wv359mLEH7Gb8y5Z7FXcOi20WJvYg+OfV1m1mLP0k9IqWXekrCpoQU1tWzt53OvnTar1aWTM+/+6hHcOmjYnVfO1Zu1eLg70SuwA0e+qSTnkM7k/Z/8vZ/Q55i5ftrSR7XiuhubuWGirzJ/0mBhGvKYF3eq9M36SSfEFvDh2nQeDY8jYtREnLQuXD6zj+TkZNZt2kbiGx81LB75MRVlN0hP+isuHTyoq6lk9W5YlGjxo6zigVqB1xaIYm8otcSy0ARY8LNuaLXwq8UXKCm3fbmsyDkTsY9gm4VpfHw8mkd/YeICFxE7jfwuwRzctZKv921m4PAXGx23N+PPdPQPolO3Ppw6lGV1jM3R3n70O9woSFR7Q6nlWAvN/n3c6d/Hg8xdJZSU69E6GRcK2IKoORO1j2CbhWlTdpx9B40D4Pa1xsMGZ4/vovjEHkZN+VOT85ttRVEUq5sIOPzKWFR7Q6nlWAvNIeGeANy6XcfvX+1BZH9PtFoNV27Usmnb9+zdr2tGoXXiehj6qDYVZcaFHe7e/ibba6sr2Lvlzzz+5FS6BQ+gMHeDqp+r6NvWkMpeHH5lLKq9odRyrIVmYIBxPu1rMwLw8tCyfO11/rb2OvV6Awt+1p24J3yaUWiduB6GPqqJoug5sPPvODk5EzJkosm+3Oy/YFAMxEx4s1U+u72Zyzu8GFtrsSe1xNZSW8/dzVg4qqsV/pB8ia8O6NhTUM7v/nqJiko9L07qjMbK2iJqzkTto5r8J3MJ188fJXrcfPy63pvLe/XcYU7kf8Tw535tdmhDDdpbMbY4TJGamtrkPo1Gw+uvW/ccLEuIam8otRxroVlbZ/yCfHVQx4+tbu9UKew/XkHcEx0JDHDl8vXaNo3rYeijWhRsXc7xnHTCYxIZOvqeHae+vpYvN/2RR/rG0C9yogUF+2hv84wt/qv08PBo1DQaDRkZGaxatUqVAES1N5RajrXQ/K7E+IihUjMzDErKjNs8Pay70hM1Z6L2UQ32bU/hwK40wqKe56kppnacx3M2UHLzHINH/ZzSWxcaWl3NHQDKb1/m0qVLdsfQ3q6MLf6lX3755YaWmJhIdXU1GRkZjB8/nt271VlDKaq9odRyrIVm0flqAPx9G/948/czbivTWXeDRtScidpHe9m3PYX9O1YQOuw5np76biM7Tl3JVQwGhax/zGHdkviGdvb4LgA2vT+FhIQEu+Nob65tzf7bLS0t5f333ychIQG9Xk9mZiaLFi3C39+/uUOtQlR7Q6nlWAvNfccqqKzSMzLKG7cO977Mfj5aogd4ceVGLddv1bV5XA9DH+1h/44V7N+xgpChkxg9zbwdZ1jU84ybldyoBfYxPrDi6WmLWbZsmd2xKHq91U0ELI4ZJyUlsWvXLhITE8nOzsbT01P1AESxN5RaYllo3qlS+HDLd7w2I4CkRUF8kVeGs7OGscN9cXbW8MGm9p8zUfsItlmYpqens2+70Y7zkX4xnDr8mcl+D29/gkJi6RIYSpfA0EbHF3/zbwAeDY9j9Gj7l0M/UBaaa9aswdXVlbS0NJOHj0oLTanVFno7c8sov6Nn8hg/pj/bGYMBThVX8T9rrnHyXLXD4noY+miLhemP7Th3b/hNo/2Bjw0jKCS2ZYHYQWsOPyiKQnJyMps3b6aqqorIyEjeeecdAgMDmz+4CSwW45MnT9os3BIUBTZ+cpmNn1yWWu1YqzX0Co5WUHDU/vFOkXMmYh/f+ENhi49ZunQpPUf8t82fOWb6UsZMX2rz8ffTmjfmVq1axWeffcb69esJCAhg6dKlzJ07l08//RQnG1cSOnyesUQikbQGBoNidWspGzduZPbs2fTu3RtPT08WLVpEcXExhw4dsjleaaEpkUgeSFpidL81fbDZp1v7+Pjg42M6Q0Wn0zF06FA+/vhjBgwY0LB9woQJTJ06lZdeesmmeB3uTSGRSCStQUueFpKSkmJ2kdu8efOYP3++ybaKCuOQ0v1F2tvbu2GfLchiLJFIHnpmzZrF5MmTG22/v+ACeHl5AcYr5B+j0+ka9tmCLMYSieShx9xwRFN4e3sTGBjIiRMniIiIAIyF+OLFi4SFhdkcg7yBJ5FIJC1k2rRprF69muLiYiorK1m2bBm9evViyJAhNmvKK2OJRCJpIbNnz0an0zF9+nSqqqoYMmQIaWlpNk9rAzmbQiKRSIRADlNIJBKJAMhiLJFIJAIgi7FEIpEIgCzGEolEIgCyGEskEokAyGIskUgkAiCLsUQikQiALMYSiUQiALIYSyQSiQD8PwJ/Z3dklZGTAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aD21fYm4DmRZ"
      },
      "source": [
        "# 6 Results analysis and Conclusion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8JNSpRJb7ofM"
      },
      "source": [
        "## Quantitative evaluation\n",
        "\n",
        "As explained earlier, to evaluate the different models **quantitatively**, we used **F1-scores** per class and their global weighted and macro values (table 1). Indeed, it is more suitable than the accuracy when the dataset is not balanced (like for us) and to account for the weakness of our models in the classification of some specific (and rare) classes. \n",
        "\n",
        "\n",
        "*Table 1 : Macro and weighted F1 scores for our 9 models*.\n",
        "\n",
        "|                                               |     Macro avg    |     Weighted   avg    |\n",
        "|-----------------------------------------------|------------------|-----------------------|\n",
        "|     Model 1:   Baseline                       |     0.50         |     0.55              |\n",
        "|     Model 2:   LSTM (fixedL+RWE)              |     0.06         |     0.10              |\n",
        "|     Model 3:   LSTM (varL+RWE)                |     0.27         |     0.33              |\n",
        "|     Model 4:   LSTM (fixedL+W2V)              |     0.11         |     0.17              |\n",
        "|     Model 5:   LSTM (varL+ W2V)               |     0.36         |     0.40              |\n",
        "|     Model 6:   LSTM (fixedL+W2V+frozen)       |     0.11         |     0.14              |\n",
        "|     Model 7:   LSTM (varL+ W2V+frozen)        |     0.69         |     0.69              |\n",
        "|     Model 8:   CamemBERT                      |     0.74         |     0.76              |\n",
        "|     Model 9:   CamemBERT with domain shift    |     0.57         |     0.62              |\n",
        "\n",
        "\n",
        "*Note :* `macro avg` *and* `weighted avg` *respectively correspond to the averages of F1 scores per class without and with considering the proportion for each label in the dataset. \n",
        "fixedL = fixed-length inputs / varL = variable-length inputs / RWE = random initialization of word-embeddings / W2V = word2vec word-embeddings / frozen = frozen word-embedding weights*\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "Model 8 (CamemBERT) is the best model followed by Model 7 (LSTM with frozen word2vec word-embeddings). Surprisingly, the baseline performs better than all the other LSTM models which means that using the words without their contexts in the description can show pretty good results in predicting the job domain. Furthermore, Model 9 shows also good results which means that the model does not seem to be very sensible to domain shift (descriptions from Onisep versus from Pôle Emploi). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GbHxN9aq9atU"
      },
      "source": [
        "\n",
        "## Qualitative evaluation \n",
        "\n",
        "We also evaluated our models **qualitatively**, giving some examples of good and bad predictions of our models.\n",
        "\n",
        "To prevent some minor changes on the predictions due to the randomness of the models, we stored in a table the predictions of the previous models before performing the qualitative evaluation. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yO-ShaRVagOG"
      },
      "source": [
        "###### To concatenate the qualitative evaluation datasets and save it to CSV (already done)\n",
        "#pred_vs_true_m1 = pred_vs_true_m1.rename(columns={\"y_pred\":\"model 1\",\n",
        "#                                                  \"label\":\"true_label\"})\n",
        "#pred_vs_true_m2 = pred_vs_true_m2.rename(columns={\"y_pred\":\"model 2\"})\n",
        "#pred_vs_true_m3 = pred_vs_true_m3.rename(columns={\"y_pred\":\"model 3\"})\n",
        "#pred_vs_true_m4 = pred_vs_true_m4.rename(columns={\"y_pred\":\"model 4\"})\n",
        "#pred_vs_true_m5 = pred_vs_true_m5.rename(columns={\"y_pred\":\"model 5\"})\n",
        "#pred_vs_true_m6 = pred_vs_true_m6.rename(columns={\"y_pred\":\"model 6\"})\n",
        "#pred_vs_true_m7 = pred_vs_true_m7.rename(columns={\"y_pred\":\"model 7\"})\n",
        "#pred_vs_true_m8 = pred_vs_true_m8.rename(columns={\"y_pred\":\"model 8\"})\n",
        "#pred_vs_true = pd.concat([pred_vs_true_m1, pred_vs_true_m2[\"model 2\"],\n",
        "#              pred_vs_true_m3[\"model 3\"],\n",
        "#              pred_vs_true_m4[\"model 4\"],\n",
        "#              pred_vs_true_m5[\"model 5\"],\n",
        "#              pred_vs_true_m6[\"model 6\"],\n",
        "#              pred_vs_true_m7[\"model 7\"],\n",
        "#              pred_vs_true_m8[\"model 8\"]], axis=1)\n",
        "#pred_vs_true.to_csv(\"pred_vs_true.csv\", index=False)\n",
        "#pred_vs_true"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WtBwLtEQlUXG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "664a06e6-3879-4d81-bd35-4866b5f2acd4"
      },
      "source": [
        "###### To download the saved database from gist\n",
        "!wget https://gist.githubusercontent.com/AQLT/1e513f8713859da30076bace09eba173/raw/5e379b699afb78c54d0c0411c4250284582d7753/pred_vs_true.csv -O pred_vs_true.csv"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-04-17 14:57:02--  https://gist.githubusercontent.com/AQLT/1e513f8713859da30076bace09eba173/raw/5e379b699afb78c54d0c0411c4250284582d7753/pred_vs_true.csv\n",
            "Resolving gist.githubusercontent.com (gist.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to gist.githubusercontent.com (gist.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 274179 (268K) [text/plain]\n",
            "Saving to: ‘pred_vs_true.csv’\n",
            "\n",
            "pred_vs_true.csv    100%[===================>] 267.75K  --.-KB/s    in 0.006s  \n",
            "\n",
            "2021-04-17 14:57:03 (41.0 MB/s) - ‘pred_vs_true.csv’ saved [274179/274179]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rYKIztPNobht"
      },
      "source": [
        "Just to give an intuition of the quality of some of the predictions, we took 2 symbolic examples (see below) from the industrial major domain (H, \"Industrie\") : \n",
        "\n",
        "1. *Observation 200* is **well predicted**. It is note very surprising because many words belonging to the vocabulary of industry are used in the description: \"fonderie\", \"forge\", \"industriels\"...\n",
        "\n",
        "2. *Observation 138* is, on the contrary **badly predicted**. Indeed, reading the description it is hard to guess that the job belongs to the industrial domain. Our two best models (7 and 8), predicted the domain \"Support à l'entreprise\" (domain M) instead. Maybe because the expressions \"ingénieur support\", \"diagnostic\" and \"accompagner le client dans la mise en place d'une solution précaunisée\" are more susceptible to make us classify this observation in domain M. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 754
        },
        "id": "bNiDjXZplfVv",
        "outputId": "9859677a-27ed-4ad5-f035-8966d814bc48"
      },
      "source": [
        "pred_vs_true = pd.read_csv(\"pred_vs_true.csv\")\n",
        "with pd.option_context('display.max_colwidth', None):\n",
        "  display(pred_vs_true.iloc[[200, 138], :])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>description</th>\n",
              "      <th>true_label</th>\n",
              "      <th>model 1</th>\n",
              "      <th>model 2</th>\n",
              "      <th>model 3</th>\n",
              "      <th>model 4</th>\n",
              "      <th>model 5</th>\n",
              "      <th>model 6</th>\n",
              "      <th>model 7</th>\n",
              "      <th>model 8</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>200</th>\n",
              "      <td>Parachève et réduit par abrasion les aspérités et rugosités des surfaces de pièces et produits industriels usinés, bruts de fonderie, de forge, ... selon les règles de sécurité et les impératifs de production (qualité, délais, ...).</td>\n",
              "      <td>H</td>\n",
              "      <td>H</td>\n",
              "      <td>H</td>\n",
              "      <td>H</td>\n",
              "      <td>H</td>\n",
              "      <td>H</td>\n",
              "      <td>H</td>\n",
              "      <td>H</td>\n",
              "      <td>H</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>138</th>\n",
              "      <td>Très réactif et efficace, l'ingénieur support intervient auprès du client en cas de problème informatique, qu'il soit logiciel ou matériel. Après avoir écarté les erreurs de manipulation ou d'installation, il doit confirmer l'incident et effectuer rapidement un diagnostic précis afin d'apporter une ou plusieurs solutions au client dans les meilleurs délais. Selon le cas, il se déplace ou règle le problème par téléphone. Expert en gestion de crise, il doit rester calme et concentré en toutes circonstances avec un seul objectif en tête : la satisfaction du client. Dans certains cas, il peut être amené à suivre et accompagner le client dans la mise en place de la solution préconisée, et parfois même à former les utilisateurs si de nouvelles fonctions ont été ajoutées ou des développements faits, par exemple. \\nPremier interlocuteur de l'utilisateur, il doit avoir le sens de l'écoute et du contact, en plus de bonnes connaissances techniques. Il participe en outre à l'amélioration des produits en faisant remonter les problèmes rencontrés par les utilisateurs à sa hiérarchie. Les débouchés sont bons pour cette profession qui requiert de l'expérience ou un diplôme de niveau bac + 5 du secteur informatique.</td>\n",
              "      <td>H</td>\n",
              "      <td>A</td>\n",
              "      <td>J</td>\n",
              "      <td>A</td>\n",
              "      <td>J</td>\n",
              "      <td>J</td>\n",
              "      <td>K</td>\n",
              "      <td>M</td>\n",
              "      <td>M</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            description  ... model 8\n",
              "200                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Parachève et réduit par abrasion les aspérités et rugosités des surfaces de pièces et produits industriels usinés, bruts de fonderie, de forge, ... selon les règles de sécurité et les impératifs de production (qualité, délais, ...).  ...       H\n",
              "138  Très réactif et efficace, l'ingénieur support intervient auprès du client en cas de problème informatique, qu'il soit logiciel ou matériel. Après avoir écarté les erreurs de manipulation ou d'installation, il doit confirmer l'incident et effectuer rapidement un diagnostic précis afin d'apporter une ou plusieurs solutions au client dans les meilleurs délais. Selon le cas, il se déplace ou règle le problème par téléphone. Expert en gestion de crise, il doit rester calme et concentré en toutes circonstances avec un seul objectif en tête : la satisfaction du client. Dans certains cas, il peut être amené à suivre et accompagner le client dans la mise en place de la solution préconisée, et parfois même à former les utilisateurs si de nouvelles fonctions ont été ajoutées ou des développements faits, par exemple. \\nPremier interlocuteur de l'utilisateur, il doit avoir le sens de l'écoute et du contact, en plus de bonnes connaissances techniques. Il participe en outre à l'amélioration des produits en faisant remonter les problèmes rencontrés par les utilisateurs à sa hiérarchie. Les débouchés sont bons pour cette profession qui requiert de l'expérience ou un diplôme de niveau bac + 5 du secteur informatique.  ...       M\n",
              "\n",
              "[2 rows x 10 columns]"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ufPqDcu-cuf2"
      },
      "source": [
        "| code domaine | libellé domaine                                                              |\n",
        "|--------------|------------------------------------------------------------------------------|\n",
        "| A            | Agriculture et   Pêche, Espaces naturels et Espaces verts, Soins aux animaux |\n",
        "| B            | Arts   et Façonnage d'ouvrages d'art                                         |\n",
        "| C            | Banque,   Assurance, Immobilier                                              |\n",
        "| D            | Commerce,   Vente et Grande distribution                                     |\n",
        "| E            | Communication,   Média et Multimédia                                         |\n",
        "| F            | Construction,   Bâtiment et Travaux publics                                  |\n",
        "| G            | Hôtellerie-Restauration,   Tourisme, Loisirs et Animation                    |\n",
        "| H            | Industrie                                                                    |\n",
        "| I            | Installation   et Maintenance                                                |\n",
        "| J            | Santé                                                                        |\n",
        "| K            | Services   à la personne et à la collectivité                                |\n",
        "| L            | Spectacle                                                                    |\n",
        "| M            | Support   à l'entreprise                                                     |\n",
        "| N            | Transport   et Logistique                                                    |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3_Ec3Y58inKE"
      },
      "source": [
        "Finally, when we compute the mean of the length of descriptions for each domain in the initial dataset, we can notice that the quality of prediction is not directly related to the length of the descriptions provided. For example, in Model 8 (our best model), the domain L has a F1 score of 1 (no error of prediction!) whereas B has a F1 score near 0 and both categories have rather long descriptions in mean (see below). "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 520
        },
        "id": "JETbZGA_jmpG",
        "outputId": "1ca7b917-a34b-4339-be3a-0f8ff347854b"
      },
      "source": [
        "# Mean of the length of descriptions for each domain in the initial dataset\n",
        "tokenizer = TweetTokenizer()\n",
        "def tokenize (text):\n",
        "  tokens = tokenizer.tokenize(text)\n",
        "  tokens = list(map(lambda x: x.lower(), tokens))\n",
        "  return tokens\n",
        "len_descript = data_baseline.copy()\n",
        "len_descript['mean_description_length'] = len_descript['description'].apply(lambda x: len(tokenize(x)))\n",
        "len_descript[['major_code', 'mean_description_length']].groupby(['major_code']).mean().sort_values(by='mean_description_length')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mean_description_length</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>major_code</th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>D</th>\n",
              "      <td>108.391892</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>G</th>\n",
              "      <td>109.390625</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>C</th>\n",
              "      <td>114.681818</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>I</th>\n",
              "      <td>115.694915</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>F</th>\n",
              "      <td>120.052083</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>A</th>\n",
              "      <td>120.297297</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>N</th>\n",
              "      <td>121.843750</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>M</th>\n",
              "      <td>128.163934</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>K</th>\n",
              "      <td>129.773399</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>H</th>\n",
              "      <td>130.379630</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>L</th>\n",
              "      <td>133.290909</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>J</th>\n",
              "      <td>133.914286</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>B</th>\n",
              "      <td>137.175000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>E</th>\n",
              "      <td>146.590909</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            mean_description_length\n",
              "major_code                         \n",
              "D                        108.391892\n",
              "G                        109.390625\n",
              "C                        114.681818\n",
              "I                        115.694915\n",
              "F                        120.052083\n",
              "A                        120.297297\n",
              "N                        121.843750\n",
              "M                        128.163934\n",
              "K                        129.773399\n",
              "H                        130.379630\n",
              "L                        133.290909\n",
              "J                        133.914286\n",
              "B                        137.175000\n",
              "E                        146.590909"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 75
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FXIpCotHn-yv"
      },
      "source": [
        "## Conclusion\n",
        "\n",
        "To conclude, both LSTM (Model 7) and CamemBERT (Model 8) seem to be efficient in guessing the domain of a job from its description. Both methods are sensible to the context of the words in the descriptions, in contrast to our baseline, and it seems to improve the results even if the baseline was not that bad.\n",
        "\n",
        "For the dataset we used, CamemBERT is the most efficient in term of weighted F1 score and Model 9 shows that it is still efficient even in presence of domain shift. Indeed, CamemBERT has the advantages of working at the sub-word units using byte-pair-encoding and thus has no Out-Of-Vocabulary problem. Thanks to the use of attention mechanisms, CamemBERT can also make some inputs more important than others; and whereas vanilla LSTM only uses left context, CamemBERT is bidirectional. However, it is also important to look at the **trade-off between performance and complexity**: our LSTM models are much faster to run and sometimes the use of a simpler model can be a good engineering option, even if it is a bit less efficient. \n",
        "\n",
        "In the future, it could be interesting to **improve the LSTM implementation** we used, making it bidirectional and adding attention mechanisms, as raised in our NLP class. However, it could slow down the process and we must keep in mind that the recurrent nature of LSTM limits the possibility to **scale the training process to more data**: we cannot parallelize LSTM easily whereas it is easier with BERT’s transformers. \n",
        "\n",
        "In addition to job descriptions, we could use **additional information** to do some further experimentations and check if necessary diplomas or conditions for exercising the activities are useful to label the job’s domain. Using **additional datasets** could also be interesting to improve the training process and to be able to predict job’s domain at a less aggregated level. We can think of online CVs or LinkedIn job offers. However, collecting such data without the consent of their authors is more prone to **ethical concerns** than using job descriptions produced by specialized agencies and voluntarily put in open data. \n"
      ]
    }
  ]
}