001_enap_grid_search_text_feature_extraction_Fernando_tcu_jurisp.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.3"
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": true,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": false,
      "toc_position": {},
      "toc_section_display": true,
      "toc_window_display": false
    },
    "colab": {
      "name": "001_enap_grid_search_text_feature_extraction_Fernando_tcu_jurisp.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Mar-celo/Enap/blob/master/001_enap_grid_search_text_feature_extraction_Fernando_tcu_jurisp.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yPB9KYHVuIjR"
      },
      "source": [
        "\n",
        "# Sklearn - NLP Text classification\n",
        "## Jurisprudências TCU\n",
        "\n",
        "Fernando Melo  \n",
        "22/7/2020\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U-mHapOPuIje"
      },
      "source": [
        "### Tokenization: processo de dividir um texto em palavras (tokens) separadas.  \n",
        "'Fui para os bosques viver de livre vontade.'   \n",
        "[ 'Fui', 'para', 'os', 'bosques', 'viver', 'de', 'livre', 'vontade', '.' ]  \n",
        "  \n",
        "### Numericalization: converter cada token em número.  \n",
        "'Fui', 'para', 'os', 'bosques', 'viver', 'de', 'livre', 'vontade', '.'   \n",
        "[ 0,1,2,3,4,5,6,7,8,9 ]  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "elyIxdYpuIjg"
      },
      "source": [
        "### Bag of words (sklearn CountVectorizer)\n",
        "Bag of words: converte cada token único em um número inteiro.  \n",
        "Problema: Ordem das palavras não importa.  \n",
        "  \n",
        "“It was the best of times”  \n",
        "“It was the worst of times”  \n",
        "“It was the age of wisdom”  \n",
        "“It was the age of foolishness”  \n",
        "   \n",
        "dicionário: ‘It’, ‘was’, ‘the’, ‘best’, ‘of’, ‘times’, ‘worst’, ‘age’, ‘wisdom’, ‘foolishness’  \n",
        "  \n",
        "cria vetores para cada frase:  \n",
        "“It was the best of times” = [1, 1, 1, 1, 1, 1, 0, 0, 0, 0]  \n",
        "“It was the worst of times” = [1, 1, 1, 0, 1, 1, 1, 0, 0, 0]  \n",
        "“It was the age of wisdom” = [1, 1, 1, 0, 1, 0, 0, 1, 1, 0]  \n",
        "“It was the age of foolishness” = [1, 1, 1, 0, 1, 0, 0, 1, 0, 1]  \n",
        "  \n",
        "n-grams:  \n",
        "gram: cada token é um 'gram':  \n",
        "‘It’, ‘was’, ‘the’, ‘best’, ‘of’, ‘times’   \n",
        "bigram : “it was”, “was the”, “the best”, “best of”, “of times”  \n",
        "trigram : “It was the\", \"was the best\", \"the best of\", \"best of times\"  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "e-gXRUccuIjh",
        "outputId": "a7449646-31eb-44a0-e19c-91ff331e6668"
      },
      "source": [
        "# No sklearn usamos CountVectorizer\n",
        "import pandas as pd\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "# lista de documentos\n",
        "text = ['O tempo é muito lento para os que esperam',\n",
        "        'Muito rápido para os que têm medo',\n",
        "        'Muito longo para os que lamentam',\n",
        "        'Muito curto para os que festejam',\n",
        "        'Mas, para os que amam e amam, o tempo é eterno.']\n",
        "# instancia o transform CountVectorizer\n",
        "vectorizer = CountVectorizer()\n",
        "# tokeniza e cria o vocabulário\n",
        "vectorizer.fit(text)\n",
        "# mostra o vocabulário criado\n",
        "print('Vocabulário: ')\n",
        "print(vectorizer.vocabulary_)\n",
        "# converte em números\n",
        "print('\\ndimensões da matrix: ')\n",
        "vector = vectorizer.transform(text)\n",
        "# # mostra as dimensões da matrix de frequência\n",
        "# print(vector.shape)\n",
        "# # mostra os documentos codificados\n",
        "# print(vector.toarray())\n",
        "# mostra os documentos codificados\n",
        "values = vectorizer.transform(text)\n",
        "feature_names = vectorizer.get_feature_names()\n",
        "print('\\n',text[0])\n",
        "pd.DataFrame(values.toarray(), columns = feature_names)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Vocabulário: \n",
            "{'tempo': 15, 'muito': 10, 'lento': 6, 'para': 12, 'os': 11, 'que': 13, 'esperam': 2, 'rápido': 14, 'têm': 16, 'medo': 9, 'longo': 7, 'lamentam': 5, 'curto': 1, 'festejam': 4, 'mas': 8, 'amam': 0, 'eterno': 3}\n",
            "\n",
            "dimensões da matrix: \n",
            "\n",
            " O tempo é muito lento para os que esperam\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>amam</th>\n",
              "      <th>curto</th>\n",
              "      <th>esperam</th>\n",
              "      <th>eterno</th>\n",
              "      <th>festejam</th>\n",
              "      <th>lamentam</th>\n",
              "      <th>lento</th>\n",
              "      <th>longo</th>\n",
              "      <th>mas</th>\n",
              "      <th>medo</th>\n",
              "      <th>muito</th>\n",
              "      <th>os</th>\n",
              "      <th>para</th>\n",
              "      <th>que</th>\n",
              "      <th>rápido</th>\n",
              "      <th>tempo</th>\n",
              "      <th>têm</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   amam  curto  esperam  eterno  festejam  lamentam  lento  longo  mas  medo  \\\n",
              "0     0      0        1       0         0         0      1      0    0     0   \n",
              "1     0      0        0       0         0         0      0      0    0     1   \n",
              "2     0      0        0       0         0         1      0      1    0     0   \n",
              "3     0      1        0       0         1         0      0      0    0     0   \n",
              "4     2      0        0       1         0         0      0      0    1     0   \n",
              "\n",
              "   muito  os  para  que  rápido  tempo  têm  \n",
              "0      1   1     1    1       0      1    0  \n",
              "1      1   1     1    1       1      0    1  \n",
              "2      1   1     1    1       0      0    0  \n",
              "3      1   1     1    1       0      0    0  \n",
              "4      0   1     1    1       0      1    0  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OhJv979iuIjj"
      },
      "source": [
        "### TF-IDF Term frequency — Inverse document frequency \n",
        "(sklearn TfidfVectorizer)  \n",
        "TF-IDF() é um método estatístico utilizado para avaliar a importância de uma palavra em determinado documento.  \n",
        "  \n",
        "TF - Frequência dos termos: fé a frequência que uma palavra aparece em um texto, divido pelo número total de palavras.  \n",
        "  \n",
        "IDF - Frequência inversa de documentos: é a pontuação de quão interessante (rara) é a palavra no texto. reduz as palavras que aparecem muito nos documentos. Sem entrar na matemática, TF-IDF são pontuações de frequência de palavras que tentam destacar palavras mais interessantes, freqüentes em um documento, mas raras entre outros documentos. Ou seja, diminui o peso das sentenças que ocorrem com mais frequência no conjunto de textos selecionados, ao mesmo tempo que aumenta o peso daqueles que ocorrem raramente."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WopXlOiUuIjk",
        "outputId": "5df5593a-97ac-459d-c0fa-299dd9c84bf7"
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "# instancia o transform CountVectorizer\n",
        "tfidf_vectorizer = TfidfVectorizer()\n",
        "# tokeniza e cria o vocabulário\n",
        "tfidf_vectorizer.fit(text)\n",
        "# mostra o vocabulário criado\n",
        "print('Vocabulário: ')\n",
        "print(tfidf_vectorizer.vocabulary_)\n",
        "print('\\nPeso de cada palavra do vocabulário: ')\n",
        "print(tfidf_vectorizer.idf_)\n",
        "# encode document\n",
        "tfidf_vector = tfidf_vectorizer.transform(text)\n",
        "# mostra as dimensões da matrix de frequência\n",
        "print('\\nDimensões da matrix: ')\n",
        "print(tfidf_vector.shape)\n",
        "# mostra os documentos codificados\n",
        "val = tfidf_vectorizer.fit_transform(text)\n",
        "feature_names = tfidf_vectorizer.get_feature_names()\n",
        "print('\\n',text[0])\n",
        "pd.DataFrame(val.toarray(), columns = feature_names)\n",
        "# print(tfidf_vector.toarray())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Vocabulário: \n",
            "{'tempo': 15, 'muito': 10, 'lento': 6, 'para': 12, 'os': 11, 'que': 13, 'esperam': 2, 'rápido': 14, 'têm': 16, 'medo': 9, 'longo': 7, 'lamentam': 5, 'curto': 1, 'festejam': 4, 'mas': 8, 'amam': 0, 'eterno': 3}\n",
            "\n",
            "Peso de cada palavra do vocabulário: \n",
            "[2.09861229 2.09861229 2.09861229 2.09861229 2.09861229 2.09861229\n",
            " 2.09861229 2.09861229 2.09861229 2.09861229 1.18232156 1.\n",
            " 1.         1.         2.09861229 1.69314718 2.09861229]\n",
            "\n",
            "Dimensões da matrix: \n",
            "(5, 17)\n",
            "\n",
            " O tempo é muito lento para os que esperam\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>amam</th>\n",
              "      <th>curto</th>\n",
              "      <th>esperam</th>\n",
              "      <th>eterno</th>\n",
              "      <th>festejam</th>\n",
              "      <th>lamentam</th>\n",
              "      <th>lento</th>\n",
              "      <th>longo</th>\n",
              "      <th>mas</th>\n",
              "      <th>medo</th>\n",
              "      <th>muito</th>\n",
              "      <th>os</th>\n",
              "      <th>para</th>\n",
              "      <th>que</th>\n",
              "      <th>rápido</th>\n",
              "      <th>tempo</th>\n",
              "      <th>têm</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.523461</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.523461</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.294909</td>\n",
              "      <td>0.249432</td>\n",
              "      <td>0.249432</td>\n",
              "      <td>0.249432</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.422325</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.500089</td>\n",
              "      <td>0.281742</td>\n",
              "      <td>0.238295</td>\n",
              "      <td>0.238295</td>\n",
              "      <td>0.238295</td>\n",
              "      <td>0.500089</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.500089</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.577488</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.577488</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.325347</td>\n",
              "      <td>0.275176</td>\n",
              "      <td>0.275176</td>\n",
              "      <td>0.275176</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.577488</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.577488</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.325347</td>\n",
              "      <td>0.275176</td>\n",
              "      <td>0.275176</td>\n",
              "      <td>0.275176</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.738612</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.369306</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.369306</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.175976</td>\n",
              "      <td>0.175976</td>\n",
              "      <td>0.175976</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.297954</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       amam     curto   esperam    eterno  festejam  lamentam     lento  \\\n",
              "0  0.000000  0.000000  0.523461  0.000000  0.000000  0.000000  0.523461   \n",
              "1  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000   \n",
              "2  0.000000  0.000000  0.000000  0.000000  0.000000  0.577488  0.000000   \n",
              "3  0.000000  0.577488  0.000000  0.000000  0.577488  0.000000  0.000000   \n",
              "4  0.738612  0.000000  0.000000  0.369306  0.000000  0.000000  0.000000   \n",
              "\n",
              "      longo       mas      medo     muito        os      para       que  \\\n",
              "0  0.000000  0.000000  0.000000  0.294909  0.249432  0.249432  0.249432   \n",
              "1  0.000000  0.000000  0.500089  0.281742  0.238295  0.238295  0.238295   \n",
              "2  0.577488  0.000000  0.000000  0.325347  0.275176  0.275176  0.275176   \n",
              "3  0.000000  0.000000  0.000000  0.325347  0.275176  0.275176  0.275176   \n",
              "4  0.000000  0.369306  0.000000  0.000000  0.175976  0.175976  0.175976   \n",
              "\n",
              "     rápido     tempo       têm  \n",
              "0  0.000000  0.422325  0.000000  \n",
              "1  0.500089  0.000000  0.500089  \n",
              "2  0.000000  0.000000  0.000000  \n",
              "3  0.000000  0.000000  0.000000  \n",
              "4  0.000000  0.297954  0.000000  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hsKTjgM7uIjk"
      },
      "source": [
        "%matplotlib inline"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JiWI2yXtuIjl",
        "outputId": "dd7fa915-e3e2-4a6f-bd5c-1bfc6aca4ef5"
      },
      "source": [
        "# # Importar bibliotecas\n",
        "from pprint import pprint\n",
        "from time import time\n",
        "import logging\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "#from sklearn.datasets import fetch_20newsgroups\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.feature_extraction.text import TfidfTransformer\n",
        "from sklearn.linear_model import SGDClassifier\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "from sklearn.pipeline import Pipeline\n",
        "\n",
        "print(__doc__)\n",
        "# Display progress logs on stdout\n",
        "logging.basicConfig(level=logging.INFO,format='%(asctime)s %(levelname)s %(message)s')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Automatically created module for IPython interactive environment\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dX814TRnuIjl"
      },
      "source": [
        "# Salvando o modelo para posterior reuso\n",
        "# from sklearn.externals import joblib\n",
        "from joblib import dump, load\n",
        "import re"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E123-k3euIjm"
      },
      "source": [
        "# Pelo que li sobre o tipo de dado utilizado pelos exemplos do sklearn: type(data) sklearn.utils.Bunch\n",
        "# Basta apresentar meus próprios dados como uma lista de strings e uma lista de rótulos (0 ou 1, no baso binário)\n",
        "\n",
        "df_tcu = pd.read_csv('tcu_jurisp_redz_v1.csv')\n",
        "df_tcu.columns = ['classe','texto']\n",
        "# em_test = pd.read_csv('data/discursosCD-SF/tfidf_sklearn/solic_ds6_tfidf12k_val.csv')\n",
        "# em_test.columns = ['classe','texto']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x33SgiDQuIjn",
        "outputId": "befa8afa-f75c-4619-fc32-5997c75dc1c2"
      },
      "source": [
        "print(df_tcu.shape)\n",
        "df_tcu.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(10263, 2)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>classe</th>\n",
              "      <th>texto</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>3</td>\n",
              "      <td>A medida cautelar do TCU que determina a suspe...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>3</td>\n",
              "      <td>A retenção de recursos pela Administração com ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>Para fins de admissibilidade de recurso de rev...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>Fotografias não têm pleno valor probatório, so...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>3</td>\n",
              "      <td>Não cabe instauração de tomada de contas espec...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   classe                                              texto\n",
              "0       3  A medida cautelar do TCU que determina a suspe...\n",
              "1       3  A retenção de recursos pela Administração com ...\n",
              "2       3  Para fins de admissibilidade de recurso de rev...\n",
              "3       3  Fotografias não têm pleno valor probatório, so...\n",
              "4       3  Não cabe instauração de tomada de contas espec..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o3LoJ49_uIjo",
        "outputId": "6da7e821-2c5e-4b61-dac4-3fcf8cc85b0e"
      },
      "source": [
        "df_tcu.classe.value_counts().plot(kind='bar')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7fa86504b3d0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD1CAYAAAC87SVQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAROUlEQVR4nO3df6zd9V3H8edr5cfmj0iRa9O1xRKtLswfZV4LZv6BWwaFmRQTXcBkawimMwF1iZp1+gduE8VEJRInSQ113bINyTZD46pYcWaZZtDL1hUKQ64MpA0/7ixjEhSFvf3jfqrHem/vuben5+7yeT6Sk/v9vj+f7/e8vyfwOl++53sOqSokSX14zXI3IEkaH0Nfkjpi6EtSRwx9SeqIoS9JHTH0JakjZyx3Aydz3nnn1caNG5e7DUlaUe6///6vVdXEXGPf0qG/ceNGpqamlrsNSVpRkjwx35iXdySpI4a+JHXE0JekjiwY+klem+S+JF9OcjjJ+1v9w0m+muRge2xu9SS5Ncl0kkNJ3jSwr+1JHm2P7afvsCRJcxnmg9yXgLdU1QtJzgQ+n+Sv2tivV9UnT5h/BbCpPS4GbgMuTnIucCMwCRRwf5K9VfXcKA5EkrSwBc/0a9YLbfXM9jjZT3NuAz7StvsCcE6StcDlwP6qOtaCfj+w9dTalyQtxlDX9JOsSnIQeJbZ4L63Dd3ULuHckuTsVlsHPDmw+ZFWm69+4nPtSDKVZGpmZmaRhyNJOpmhQr+qXqmqzcB6YEuSHwLeB7wB+HHgXOC9o2ioqnZV1WRVTU5MzPndAknSEi3qy1lV9fUknwW2VtXvt/JLSf4M+LW2fhTYMLDZ+lY7Clx6Qv3vl9DzKdm48zPjfsolefzmty93C5JehYa5e2ciyTlt+XXA24CvtOv0JAlwFfBg22Qv8K52F88lwPNV9RRwN3BZktVJVgOXtZokaUyGOdNfC+xJsorZN4k7q+ovk/xdkgkgwEHgF9v8fcCVwDTwInAtQFUdS/JB4ECb94GqOja6Q5EkLWTB0K+qQ8BFc9TfMs/8Aq6fZ2w3sHuRPUqSRsRv5EpSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1ZMHQT/LaJPcl+XKSw0ne3+oXJLk3yXSSP09yVquf3dan2/jGgX29r9UfSXL56TooSdLchjnTfwl4S1X9KLAZ2JrkEuD3gFuq6vuB54Dr2vzrgOda/ZY2jyQXAlcDbwS2An+SZNUoD0aSdHILhn7NeqGtntkeBbwF+GSr7wGuasvb2jpt/K1J0up3VNVLVfVVYBrYMpKjkCQNZahr+klWJTkIPAvsB/4Z+HpVvdymHAHWteV1wJMAbfx54LsH63NsM/hcO5JMJZmamZlZ/BFJkuY1VOhX1StVtRlYz+zZ+RtOV0NVtauqJqtqcmJi4nQ9jSR1aVF371TV14HPAj8BnJPkjDa0Hjjalo8CGwDa+HcB/zpYn2MbSdIYDHP3zkSSc9ry64C3AQ8zG/4/26ZtB+5qy3vbOm3876qqWv3qdnfPBcAm4L5RHYgkaWFnLDyFtcCedqfNa4A7q+ovkzwE3JHkt4EvAbe3+bcDH00yDRxj9o4dqupwkjuBh4CXgeur6pXRHo4k6WQWDP2qOgRcNEf9Mea4+6aq/gP4uXn2dRNw0+LblCSNgt/IlaSOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOjLMD65J89q48zPL3cJQHr/57cvdgvQtwTN9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcWDP0kG5J8NslDSQ4n+ZVW/60kR5McbI8rB7Z5X5LpJI8kuXygvrXVppPsPD2HJEmazzDfyH0Z+NWq+mKS7wTuT7K/jd1SVb8/ODnJhcDVwBuB1wN/m+QH2vCHgLcBR4ADSfZW1UOjOBBJ0sIWDP2qegp4qi3/W5KHgXUn2WQbcEdVvQR8Nck0sKWNTVfVYwBJ7mhzDX1JGpNFXdNPshG4CLi3lW5IcijJ7iSrW20d8OTAZkdabb66JGlMhg79JN8BfAp4T1V9A7gN+D5gM7P/JfAHo2goyY4kU0mmZmZmRrFLSVIzVOgnOZPZwP9YVX0aoKqeqapXquqbwJ/yv5dwjgIbBjZf32rz1f+PqtpVVZNVNTkxMbHY45EkncQwd+8EuB14uKr+cKC+dmDazwAPtuW9wNVJzk5yAbAJuA84AGxKckGSs5j9sHfvaA5DkjSMYe7eeTPwTuCBJAdb7TeAa5JsBgp4HHg3QFUdTnInsx/QvgxcX1WvACS5AbgbWAXsrqrDIzwWSdIChrl75/NA5hjad5JtbgJumqO+72TbSZJOL7+RK0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSR4b5cpakMdm48zPL3cJQHr/57cvdgpbIM31J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOLBj6STYk+WySh5IcTvIrrX5ukv1JHm1/V7d6ktyaZDrJoSRvGtjX9jb/0STbT99hSZLmMsyZ/svAr1bVhcAlwPVJLgR2AvdU1SbgnrYOcAWwqT12ALfB7JsEcCNwMbAFuPH4G4UkaTwWDP2qeqqqvtiW/w14GFgHbAP2tGl7gKva8jbgIzXrC8A5SdYClwP7q+pYVT0H7Ae2jvRoJEkntahr+kk2AhcB9wJrquqpNvQ0sKYtrwOeHNjsSKvNV5ckjcnQoZ/kO4BPAe+pqm8MjlVVATWKhpLsSDKVZGpmZmYUu5QkNUOFfpIzmQ38j1XVp1v5mXbZhvb32VY/CmwY2Hx9q81X/z+qaldVTVbV5MTExGKORZK0gGHu3glwO/BwVf3hwNBe4PgdONuBuwbq72p38VwCPN8uA90NXJZkdfsA97JWkySNyTD/Y/Q3A+8EHkhysNV+A7gZuDPJdcATwDva2D7gSmAaeBG4FqCqjiX5IHCgzftAVR0byVFIkoayYOhX1eeBzDP81jnmF3D9PPvaDexeTIOSpNHxG7mS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0JekjgzzjVxJWnE27vzMcrcwlMdvfvtYn88zfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1ZMHQT7I7ybNJHhyo/VaSo0kOtseVA2PvSzKd5JEklw/Ut7badJKdoz8USdJChjnT/zCwdY76LVW1uT32ASS5ELgaeGPb5k+SrEqyCvgQcAVwIXBNmytJGqMFf1q5qj6XZOOQ+9sG3FFVLwFfTTINbGlj01X1GECSO9rchxbdsSRpyU7lmv4NSQ61yz+rW20d8OTAnCOtNl9dkjRGSw3924DvAzYDTwF/MKqGkuxIMpVkamZmZlS7lSSxxNCvqmeq6pWq+ibwp/zvJZyjwIaBqetbbb76XPveVVWTVTU5MTGxlPYkSfNYUugnWTuw+jPA8Tt79gJXJzk7yQXAJuA+4ACwKckFSc5i9sPevUtvW5K0FAt+kJvkE8ClwHlJjgA3Apcm2QwU8DjwboCqOpzkTmY/oH0ZuL6qXmn7uQG4G1gF7K6qwyM/GknSSQ1z9841c5RvP8n8m4Cb5qjvA/YtqjtJ0kj5jVxJ6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0JekjiwY+kl2J3k2yYMDtXOT7E/yaPu7utWT5NYk00kOJXnTwDbb2/xHk2w/PYcjSTqZYc70PwxsPaG2E7inqjYB97R1gCuATe2xA7gNZt8kgBuBi4EtwI3H3ygkSeOzYOhX1eeAYyeUtwF72vIe4KqB+kdq1heAc5KsBS4H9lfVsap6DtjP/38jkSSdZku9pr+mqp5qy08Da9ryOuDJgXlHWm2+uiRpjE75g9yqKqBG0AsASXYkmUoyNTMzM6rdSpJYeug/0y7b0P4+2+pHgQ0D89a32nz1/6eqdlXVZFVNTkxMLLE9SdJclhr6e4Hjd+BsB+4aqL+r3cVzCfB8uwx0N3BZktXtA9zLWk2SNEZnLDQhySeAS4Hzkhxh9i6cm4E7k1wHPAG8o03fB1wJTAMvAtcCVNWxJB8EDrR5H6iqEz8cliSdZguGflVdM8/QW+eYW8D18+xnN7B7Ud1JkkbKb+RKUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdeSUQj/J40keSHIwyVSrnZtkf5JH29/VrZ4ktyaZTnIoyZtGcQCSpOGN4kz/p6pqc1VNtvWdwD1VtQm4p60DXAFsao8dwG0jeG5J0iKcjss724A9bXkPcNVA/SM16wvAOUnWnobnlyTN41RDv4C/SXJ/kh2ttqaqnmrLTwNr2vI64MmBbY+0miRpTM44xe1/sqqOJvkeYH+SrwwOVlUlqcXssL157AA4//zzT7E9SdKgUzrTr6qj7e+zwF8AW4Bnjl+2aX+fbdOPAhsGNl/faifuc1dVTVbV5MTExKm0J0k6wZJDP8m3J/nO48vAZcCDwF5ge5u2HbirLe8F3tXu4rkEeH7gMpAkaQxO5fLOGuAvkhzfz8er6q+THADuTHId8ATwjjZ/H3AlMA28CFx7Cs8tSVqCJYd+VT0G/Ogc9X8F3jpHvYDrl/p8kqRT5zdyJakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjoy9tBPsjXJI0mmk+wc9/NLUs/GGvpJVgEfAq4ALgSuSXLhOHuQpJ6N+0x/CzBdVY9V1X8CdwDbxtyDJHUrVTW+J0t+FthaVb/Q1t8JXFxVNwzM2QHsaKs/CDwytgaX7jzga8vdxKuIr+do+XqOzkp5Lb+3qibmGjhj3J0spKp2AbuWu4/FSDJVVZPL3cerha/naPl6js6r4bUc9+Wdo8CGgfX1rSZJGoNxh/4BYFOSC5KcBVwN7B1zD5LUrbFe3qmql5PcANwNrAJ2V9XhcfZwmqyoy1ErgK/naPl6js6Kfy3H+kGuJGl5+Y1cSeqIoS9JHTH0Jakj33L36a8ESd7A7DeJ17XSUWBvVT28fF1J//PP5jrg3qp6YaC+tar+evk6W5mSbAGqqg60n4zZCnylqvYtc2tL5pn+IiV5L7M/HxHgvvYI8Al/QG60kly73D2sJEl+GbgL+CXgwSSDP3HyO8vT1cqV5EbgVuC2JL8L/DHw7cDOJL+5rM2dAu/eWaQk/wS8sar+64T6WcDhqtq0PJ29+iT5l6o6f7n7WCmSPAD8RFW9kGQj8Engo1X1R0m+VFUXLWuDK0x7PTcDZwNPA+ur6htJXsfsf0n9yLI2uERe3lm8bwKvB544ob62jWkRkhyabwhYM85eXgVec/ySTlU9nuRS4JNJvpfZ11OL83JVvQK8mOSfq+obAFX170lW7L/rhv7ivQe4J8mjwJOtdj7w/cAN826l+awBLgeeO6Ee4B/H386K9kySzVV1EKCd8f80sBv44eVtbUX6zyTfVlUvAj92vJjku1jBJ3he3lmCJK9h9meiBz/IPdDOCrQISW4H/qyqPj/H2Mer6ueXoa0VKcl6Zs9On55j7M1V9Q/L0NaKleTsqnppjvp5wNqqemAZ2jplhr4kdcS7dySpI4a+JHXE0Jekjhj6ktQRQ1+SOvLfc3vN5iVocj4AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qgpbeHBWuIjp"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "X_train, X_test, y_train, y_test = train_test_split(df_tcu.texto, df_tcu.classe, test_size=0.2, \n",
        "                                                    stratify=df_tcu.classe,random_state=42)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GCc9WapBuIjp",
        "outputId": "916cef2b-fa5b-449c-dff0-0170c408ba84"
      },
      "source": [
        "X_train[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3965    Não se pode exigir como condição de habilitaçã...\n",
              "1976    A Administração, em respeito à transparência e...\n",
              "1253    O encaminhamento de representação ao TCU, sem ...\n",
              "4818    Havendo alteração nos pressupostos de fato que...\n",
              "2126    No pregão, o exame do registro da intenção de ...\n",
              "3107    No gerenciamento de ata de registro de preços,...\n",
              "3739    É inconstitucional o Decreto 2.745/1998 e o ar...\n",
              "1989    As entidades do Sistema S não se sujeitam à es...\n",
              "9522    A atuação de gestor com base em parecer técnic...\n",
              "7262    Há impossibilidade da percepção de anuênios pe...\n",
              "Name: texto, dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "49KmnHAnuIjq",
        "outputId": "8c582fa7-7c77-4f8e-ecba-b961b586f543"
      },
      "source": [
        "print('TREINO : ',y_train.shape,'\\n', y_train.value_counts(normalize=True),'\\n')\n",
        "print('TESTE  : ',y_test.shape,'\\n', y_test.value_counts(normalize=True))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-07-24 03:31:48,795 INFO Note: NumExpr detected 16 cores but \"NUMEXPR_MAX_THREADS\" not set, so enforcing safe limit of 8.\n",
            "2020-07-24 03:31:48,796 INFO NumExpr defaulting to 8 threads.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "TREINO :  (8210,) \n",
            " 0    0.337881\n",
            "1    0.265286\n",
            "2    0.223873\n",
            "3    0.172960\n",
            "Name: classe, dtype: float64 \n",
            "\n",
            "TESTE  :  (2053,) \n",
            " 0    0.338042\n",
            "1    0.265465\n",
            "2    0.223575\n",
            "3    0.172918\n",
            "Name: classe, dtype: float64\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "koVQ58x-uIjq",
        "outputId": "df6f76ca-63b4-42cc-dd7e-e16052ab2403"
      },
      "source": [
        "text_train = list(X_train)\n",
        "text_test = list(X_test)\n",
        "text_train[:5]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Não se pode exigir como condição de habilitação de licitante o pagamento de valor superior ao custo efetivo de reprodução gráfica do edital, sob pena de enriquecimento ilícito da Administração contratante.',\n",
              " 'A Administração, em respeito à transparência e à motivação dos atos administrativos, deve explicitar as razões para a admissão ou vedação à participação de consórcios de empresas quando da contratação de objetos de maior vulto e complexidade.',\n",
              " 'O encaminhamento de representação ao TCU, sem que o órgão fiscalizado tenha oportunidade de se manifestar, caracteriza <i>erro in procedendo</i>, prejudicando o mérito da decisão.',\n",
              " 'Havendo alteração nos pressupostos de fato que fundamentaram decisão judicial transitada em julgado que reconheceu a servidor o direito a determinado acréscimo remuneratório, a eficácia do julgado cessa de imediato, independentemente de ação rescisória.',\n",
              " 'No pregão, o exame do registro da intenção de recurso deve limitar-se à verificação dos requisitos de sucumbência, tempestividade, legitimidade, interesse e motivação, não podendo o mérito do recurso ser julgado previamente à apresentação das razões e contrarrazões recursais.']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EkzUQpAVuIjr",
        "outputId": "1325ace0-5281-446a-cdc3-12622290771a"
      },
      "source": [
        "target_train = list(y_train)\n",
        "target_test = list(y_test)\n",
        "print(target_test[:5])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[3, 3, 0, 2, 0]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J5XxDAoXuIjr"
      },
      "source": [
        "# Define a pipeline combining a text feature extractor with a simple classifier\n",
        "pipeline = Pipeline([\n",
        "    ('vect', CountVectorizer()),\n",
        "    ('tfidf', TfidfTransformer()),\n",
        "    ('clf', SGDClassifier(loss='log')), # default hinge. to get proba=log or modified_huber\n",
        "])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "112OtKtnuIjr"
      },
      "source": [
        "# uncommenting more parameters will give better exploring power but will\n",
        "# increase processing time in a combinatorial way\n",
        "parameters = {\n",
        "    'vect__max_df': (0.5, 0.75, 1.0),\n",
        "    #\n",
        "    'vect__max_features': (None, 5000, 10000, 50000),\n",
        "    'vect__ngram_range': ((1, 1), (1, 2)),  # unigrams or bigrams\n",
        "    #\n",
        "    'tfidf__use_idf': (True, False),\n",
        "    #\n",
        "    'tfidf__norm': ('l1', 'l2'),\n",
        "    'clf__max_iter': (20,),\n",
        "    'clf__alpha': (0.00001, 0.000001),\n",
        "    'clf__penalty': ('l2', 'elasticnet'),\n",
        "    #\n",
        "    'clf__max_iter': (10, 50, 80),\n",
        "}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r3fXywFauIjs",
        "outputId": "f8cc904e-2d60-4d25-a201-2b7564ae865c"
      },
      "source": [
        "if __name__ == \"__main__\":\n",
        "    # multiprocessing requires the fork to happen in a __main__ protected\n",
        "    # block\n",
        "\n",
        "    # find the best parameters for both the feature extraction and the\n",
        "    # classifier\n",
        "    grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1,cv=3)\n",
        "\n",
        "    print(\"Performing grid search...\")\n",
        "    print(\"pipeline:\", [name for name, _ in pipeline.steps])\n",
        "    print(\"parameters:\")\n",
        "    pprint(parameters)\n",
        "    t0 = time()\n",
        "    \n",
        "    grid_search.fit(text_train, target_train)\n",
        "    print(\"done in %0.3fs\" % (time() - t0))\n",
        "    print()\n",
        "\n",
        "    print(\"Best score: %0.3f\" % grid_search.best_score_)\n",
        "    print(\"Best parameters set:\")\n",
        "    best_parameters = grid_search.best_estimator_.get_params()\n",
        "    for param_name in sorted(parameters.keys()):\n",
        "        print(\"\\t%s: %r\" % (param_name, best_parameters[param_name]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Performing grid search...\n",
            "pipeline: ['vect', 'tfidf', 'clf']\n",
            "parameters:\n",
            "{'clf__alpha': (1e-05, 1e-06),\n",
            " 'clf__max_iter': (10, 50, 80),\n",
            " 'clf__penalty': ('l2', 'elasticnet'),\n",
            " 'tfidf__norm': ('l1', 'l2'),\n",
            " 'tfidf__use_idf': (True, False),\n",
            " 'vect__max_df': (0.5, 0.75, 1.0),\n",
            " 'vect__max_features': (None, 5000, 10000, 50000),\n",
            " 'vect__ngram_range': ((1, 1), (1, 2))}\n",
            "Fitting 3 folds for each of 1152 candidates, totalling 3456 fits\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 6 concurrent workers.\n",
            "[Parallel(n_jobs=-1)]: Done  38 tasks      | elapsed:    3.7s\n",
            "[Parallel(n_jobs=-1)]: Done 188 tasks      | elapsed:   16.1s\n",
            "[Parallel(n_jobs=-1)]: Done 438 tasks      | elapsed:   38.3s\n",
            "[Parallel(n_jobs=-1)]: Done 788 tasks      | elapsed:  1.1min\n",
            "[Parallel(n_jobs=-1)]: Done 1238 tasks      | elapsed:  1.8min\n",
            "[Parallel(n_jobs=-1)]: Done 1788 tasks      | elapsed:  2.6min\n",
            "[Parallel(n_jobs=-1)]: Done 2438 tasks      | elapsed:  3.6min\n",
            "[Parallel(n_jobs=-1)]: Done 3188 tasks      | elapsed:  4.7min\n",
            "[Parallel(n_jobs=-1)]: Done 3456 out of 3456 | elapsed:  5.2min finished\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "done in 312.232s\n",
            "\n",
            "Best score: 0.950\n",
            "Best parameters set:\n",
            "\tclf__alpha: 1e-06\n",
            "\tclf__max_iter: 50\n",
            "\tclf__penalty: 'l2'\n",
            "\ttfidf__norm: 'l2'\n",
            "\ttfidf__use_idf: True\n",
            "\tvect__max_df: 0.75\n",
            "\tvect__max_features: None\n",
            "\tvect__ngram_range: (1, 2)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bMa5k6F2uIjs",
        "outputId": "0f226d28-8891-4f8a-c1a1-7ea9acc1ce2f"
      },
      "source": [
        "grid_search.best_estimator_.predict(text_test)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([3, 3, 0, ..., 0, 3, 0])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "05lCghFguIjt",
        "outputId": "9505b487-4519-4c6f-dac1-990d84d413a2"
      },
      "source": [
        "prob_preds= grid_search.best_estimator_.predict_proba(text_test)\n",
        "prob_preds[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[3.75158911e-05, 3.98210087e-05, 1.94968263e-03, 9.97972980e-01],\n",
              "       [1.18136667e-01, 3.91944536e-04, 3.37293969e-02, 8.47741992e-01],\n",
              "       [9.90769586e-01, 4.07382143e-03, 3.78146696e-03, 1.37512539e-03],\n",
              "       [4.75453611e-04, 4.20663829e-03, 9.92248715e-01, 3.06919323e-03],\n",
              "       [9.94208630e-01, 5.25951236e-03, 2.03688473e-04, 3.28169014e-04],\n",
              "       [1.01842777e-01, 8.11754840e-01, 7.37871789e-04, 8.56645112e-02],\n",
              "       [9.90892274e-01, 7.90405243e-04, 7.87286206e-03, 4.44458569e-04],\n",
              "       [1.79736193e-03, 8.32924497e-03, 6.20394553e-03, 9.83669448e-01],\n",
              "       [9.96228137e-01, 1.59523842e-03, 6.33315499e-04, 1.54330896e-03],\n",
              "       [1.01685331e-03, 9.93022118e-01, 6.62553263e-04, 5.29847524e-03]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XL3XAffluIjt"
      },
      "source": [
        "df_test = pd.DataFrame(columns=['texto','predicted', 'prob_0','prob_1','prob_2','prob_3'])\n",
        "df_test['texto'] = text_test\n",
        "df_test['predicted'] = grid_search.best_estimator_.predict(text_test)\n",
        "df_test['prob_0'] = prob_preds[0:,0].tolist()\n",
        "df_test['prob_1'] = prob_preds[0:,1].tolist()\n",
        "df_test['prob_2'] = prob_preds[0:,2].tolist()\n",
        "df_test['prob_3'] = prob_preds[0:,3].tolist()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EYt2VL5muIjt",
        "outputId": "0b4aea1d-760a-499c-b1c3-617ec57a9682"
      },
      "source": [
        "print(df_test.shape)\n",
        "df_test.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(2053, 6)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>texto</th>\n",
              "      <th>predicted</th>\n",
              "      <th>prob_0</th>\n",
              "      <th>prob_1</th>\n",
              "      <th>prob_2</th>\n",
              "      <th>prob_3</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Não cabe ao TCU, a pedido do responsável, real...</td>\n",
              "      <td>3</td>\n",
              "      <td>0.000038</td>\n",
              "      <td>0.000040</td>\n",
              "      <td>0.001950</td>\n",
              "      <td>0.997973</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Não há amparo legal para condicionar o julgame...</td>\n",
              "      <td>3</td>\n",
              "      <td>0.118137</td>\n",
              "      <td>0.000392</td>\n",
              "      <td>0.033729</td>\n",
              "      <td>0.847742</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Considera-se prejudicada, por perda de objeto,...</td>\n",
              "      <td>0</td>\n",
              "      <td>0.990770</td>\n",
              "      <td>0.004074</td>\n",
              "      <td>0.003781</td>\n",
              "      <td>0.001375</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>O TCU pode desconsiderar a personalidade juríd...</td>\n",
              "      <td>2</td>\n",
              "      <td>0.000475</td>\n",
              "      <td>0.004207</td>\n",
              "      <td>0.992249</td>\n",
              "      <td>0.003069</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>É ilegal o pagamento de vantagem pessoal, com ...</td>\n",
              "      <td>0</td>\n",
              "      <td>0.994209</td>\n",
              "      <td>0.005260</td>\n",
              "      <td>0.000204</td>\n",
              "      <td>0.000328</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                               texto  predicted    prob_0  \\\n",
              "0  Não cabe ao TCU, a pedido do responsável, real...          3  0.000038   \n",
              "1  Não há amparo legal para condicionar o julgame...          3  0.118137   \n",
              "2  Considera-se prejudicada, por perda de objeto,...          0  0.990770   \n",
              "3  O TCU pode desconsiderar a personalidade juríd...          2  0.000475   \n",
              "4  É ilegal o pagamento de vantagem pessoal, com ...          0  0.994209   \n",
              "\n",
              "     prob_1    prob_2    prob_3  \n",
              "0  0.000040  0.001950  0.997973  \n",
              "1  0.000392  0.033729  0.847742  \n",
              "2  0.004074  0.003781  0.001375  \n",
              "3  0.004207  0.992249  0.003069  \n",
              "4  0.005260  0.000204  0.000328  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eWbTgpR8uIju"
      },
      "source": [
        "df_test.to_csv('tcu_tfidf_all_predictions_probs.csv', index=None)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nk1cy3uYuIju"
      },
      "source": [
        "# target_test"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vNaVlQQmuIju",
        "outputId": "206866c7-663f-47e7-9135-eab59bb8f281"
      },
      "source": [
        "grid_search.best_estimator_.predict(text_test) == target_test"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ True,  True,  True, ...,  True,  True,  True])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Llu0ISkquIjv",
        "outputId": "3a7f1e10-d50b-4a53-9a45-1cb071fdf837"
      },
      "source": [
        "sum(grid_search.best_estimator_.predict(text_test) == target_test)/len(target_test)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9590842669264491"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "kVCkPD3vuIjv",
        "outputId": "693149fa-6410-48fd-d099-bc10ab257549"
      },
      "source": [
        "grid_search.best_estimator_.score(text_test, target_test)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9590842669264491"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mji-oo0puIjw"
      },
      "source": [
        "p = grid_search.best_estimator_.predict(text_test)\n",
        "y = target_test"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vk3QMKSuuIjw",
        "outputId": "5fcec646-f983-413f-af73-7e385dcb437c"
      },
      "source": [
        "from sklearn.metrics import f1_score\n",
        "f1_score(y, p, average='weighted') "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9589655379737931"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cSRyvt79uIjx",
        "outputId": "ea37b3dd-8a77-4d4c-d9dc-f967158f823c"
      },
      "source": [
        "# Gerando Matriz de Confusão\n",
        "# https://stackoverflow.com/questions/19233771/sklearn-plot-confusion-matrix-with-labels/48018785\n",
        "\n",
        "from sklearn.metrics import confusion_matrix\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.cm as cm\n",
        "\n",
        "labels = [0,1,2,3]\n",
        "#tlabels = list(target_test.map({0:'direito', 1:'economia', 2:'social', 3: ''}))\n",
        "cm = confusion_matrix(y, p, labels)\n",
        "print(cm)\n",
        "fig = plt.figure()\n",
        "ax = fig.add_subplot(111)\n",
        "cax = ax.matshow(cm)\n",
        "\n",
        "plt.title('Confusion matrix of the classifier')\n",
        "fig.colorbar(cax)\n",
        "ax.set_xticklabels([''] + labels)\n",
        "ax.set_yticklabels([''] + labels)\n",
        "plt.xlabel('Predicted')\n",
        "plt.ylabel('True')\n",
        "for x in range(len(cm)):\n",
        "    for z in range(len(cm)):\n",
        "        plt.annotate(str(cm[x][z]), xy=(z, x),\n",
        "                    horizontalalignment='center',\n",
        "                    verticalalignment='center')\n",
        "plt.cm.Blues\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[680   6   5   3]\n",
            " [  2 537   4   2]\n",
            " [  5  11 425  18]\n",
            " [  5   5  18 327]]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/opt/conda/lib/python3.8/site-packages/sklearn/utils/validation.py:68: FutureWarning: Pass labels=[0, 1, 2, 3] as keyword args. From version 0.25 passing these as positional arguments will result in an error\n",
            "  warnings.warn(\"Pass {} as keyword args. From version 0.25 \"\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAATgAAAEQCAYAAAAkgGgxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZwcdZ3/8dd77sl9zOROIISQJQQIIYQshwYQISjCzwNFQHDReK2rwq6y/NZFUPcBu6sBj1UjoCAqIuAPlAhiAEOWHCQhhNwJR8h935lkZro/vz/qO6ET5uie6Z6ervk8H496THVVddWneno+872qSmaGc87FUVG+A3DOuVzxBOeciy1PcM652PIE55yLLU9wzrnY8gTnnIstT3DHkFQp6Y+S9kj6fRv2c42kv2QztnyRdL6klTnYb8aftaQXJH0m27Ecc4wbJM3K4f7/LOn6lNffkbRd0mZJwyTtl1Scq+N3JiX5DqC1JH0SuAn4O2AfsAj4rpm19Yv5UaA/0NfM6lu7EzP7NfDrNsaSc5IMGGlma5raxsxeBEbl4PDNftaSvgWcaGbX5uDYeWNmkxvmJQ0DbgaOM7OtYXG3vAQWQwVZgpN0E3A38B9EfyDDgP8BrsjC7o8DVrUlucWJpFz+E/TPOvru7khJbq2W499VYTKzgpqAnsB+4GPNbFNOlAA3huluoDysmwSsJ/qvuRXYBHw6rLsdqAXqwjFuBL4FPJSy7+MBA0rC6xuAN4hKkW8C16Qsn5XyvnOAl4E94ec5KeteAL4N/G/Yz1+AqibOrSH+r6fEfyVwGbAK2AncmrL9BGA2sDts+yOgLKybGc7lQDjfj6fs/xvAZuBXDcvCe0aEY4wLrwcB24BJTcR7cji/3cBS4ENNfdbHvO/SY9a/ms5nBUwEXgrHe7WpuMK2Q4HHQ/w7gB818bu7B1gH7AUWAOcf8/nOD+u2AN8PyyuAh8J+d4ffef+Uc/gM8D6gBkiGc/wl7/5+9QTuC7+7DcB3gOKUOP8XmBqO8518/312tCnvAWQccPTFr2/4AjSxzR3AHKAfUB2+8N8O6yaF998BlBIlhoNA77D+Wxyd0I59feQLCHQNX+xRYd1A4JSUL9+sMN8H2AVcF953dXjdN6x/AXgdOAmoDK/vbOLcGuL/9xD/Z8Mf6G+A7sAp4Y9meNj+TKI/+pIQ+3Lgqyn7M6Jq4LH7v4voH0UlKQkubPNZYBnQBXgG+O8mYi0F1gC3AmXAhURJaVRjn20j73/X+uY+K2Aw0R/6ZUS1k4vD6+pG9l1MlACnht9jBXDesb+78PpaoG/4DG8mSvwVYd1s4Low3w2YGOY/B/wxfEbF4ffQI+UcPpPyead+tsdzdIL7A/CzEGM/YB7wuZQ464Evh9gq8/332dGmQqyi9gW2W/PVmmuAO8xsq5ltIyotXJeyvi6srzOz6UT/PVvbxpQExkiqNLNNZra0kW0+AKw2s1+ZWb2Z/RZYAVyess0vzGyVmdUAjwBjmzlmHVF7Yx3wMFAF3GNm+8LxlwGnA5jZAjObE477FtEfy3vTOKfbzOxwiOcoZvZzosQ1lyip/98m9jOR6I/+TjOrNbPngD8RJfi2aOqzuhaYbmbTzSxpZs8Sla4ua2QfE4hKn/9iZgfM7JA10X5rZg+Z2Y7wGX6PKPE3fF/qgBMlVZnZfjObk7K8L9E/j0T4PezN5CQl9Q+xfzXEuJUoIX8iZbONZvbDENu7fledXSEmuB1AVQvtDYOAtSmv14ZlR/ZxTII8SCsads3sAFG17vPAJklPSfq7NOJpiGlwyuvNGcSzw8wSYb7hS70lZX1Nw/slnSTpT6GHbi9Ru2VVM/sG2GZmh1rY5ufAGOCHZna4iW0GAevMLJmy7Njzbo2mPqvjgI9J2t0wAecRJeFjDQXWtvCPEgBJ/yxpeejt3U1UbWz4DG8kKk2ukPSypA+G5b8iKt0+LGmjpP+UVJrheR5HVArelHI+PyMqyTVYl+E+O5VCTHCzgcNE7U5N2Uj05WgwLCxrjQNE1YwGA1JXmtkzZnYx0R/RCqI//JbiaYhpQytjysRPiOIaaWY9iKqLauE9zd5iRlI3onbN+4BvSerTxKYbgaGSUr9nmZx3pre6WQf8ysx6pUxdzezOJrYd1lLDvKTzido7ryJqxuhF1I4qADNbbWZXEyWdu4BHJXUNtYPbzWw0UfvrB4FPteJ8DhO1MTacTw8zOyVlG78dUDMKLsGZ2R6i9qcfS7pSUhdJpZImS/rPsNlvgX+TVC2pKmz/UCsPuQh4Txif1BP414YVkvpLukJSV6Iv4n6i6t2xpgMnSfqkpBJJHwdGE1XXcq07UTvh/lC6/MIx67cAJ2S4z3uA+Wb2GeAp4KdNbDeXqIT19fA7mkRULX84zeNsAY4/JkE25yHgckmXSCqWVCFpkqQhjWw7j6jh/k5JXcO25zayXXeidq5tQImkfwd6NKyUdK2k6lBK3R0WJyVdIOnUMJ5tL1GVtbHvRpPMbBNRJ8r3JPWQVCRphKSWmhhcUHAJDiC0g9wE/BvRF28d8I/A/wubfIeo7WUx8BqwMCxrzbGeBX4X9rWAo5NSUYhjI1HP4nt5dwLBzHYQ/Qe/maiK/XXgg2a2vTUxZeifgU8SNe7/nOhcUn0LeCBUga5qaWeSriDq6Gk4z5uAcZKuOXZbM6slSmiTge1EQ3k+ZWYr0oy9YfDvDkkLW9rYzNYRDRW6lXe+F/9CI9/zUMW/HDgReJuo5/jjjez2GeBpoh7qtcAhjq4WXgoslbSfKPF/IrSFDQAeJUpuy4G/EVVbM/Upog6aZUQdU4/SeJXbNUJmnauEK+lSoi9iMXBvE9WXgiPpfqIkutXMxuQ7nmyRNBR4kGi8owHTzOye/EbVdpIqiIbplBP1gD5qZrflN6r46VQJLlQXVhENH1hPNDbpajNbltfAskDSe4iqyA/GLMENBAaa2UJJ3YlK0VcW+u9MkoCuZrY/dD7MAr6S0gvrsqAgq6htMAFYY2ZvhOrTw2Tn6oe8M7OZRNXkWAlDbxaG+X1E1b229sLmnUX2h5elYeo8pY120tkS3GCObj9ZTwz+WDoLSccDZxB1XhS80BGyiOiKlGfNLBbn1ZF0tgTnClQYmvIY0aDXjAbMdlRhAPBYYAgwQVJsmhY6is6W4DYQDfBsMIT2GYvm2iC0UT0G/NrMHs93PNlmZruB54l6ZF0WdbYE9zIwUtJwSWVEl7w8meeYXDNCY/x9wHIz+36+48mWMEazV5ivJOr4Snf4jEtTp0pw4bKcfyQa27QceKSJa0cLjqTfEl3lMUrSekk35jumLDmX6DriCyUtClNj15YWmoHA85IWE/3jfdbM2mPgd6fSqYaJOOc6l05VgnPOdS6e4JxzseUJzjkXW57gnHOx1WkTnKQp+Y4hF/y8Ck+czy3fOm2CA+L6pfLzKjxxPre86swJzjkXcx1qHFxVn2I7fmimt61vnW07ElT3bZ+Hh69a3KXljbKkjsOUUt5ux2svcT0vaN9zO8QBau1wS7esb9YlF3S1HTsTLW8ILFh8+Bkzy9slaB3qQbHHDy1l3jNDW96wwFwy+Ix8h+Ay1YH+8WfTXJvR5n3s2Jlg3jPD0tq2eODqlh5wlFMdKsE55zo+A5KZPV4ibzzBOecyYhh1ll4VNd88wTnnMuYlOOdcLBlGokDaKD3BOecyliyQx0d4gnPOZcSAhCc451xceQnOORdLBtR5G5xzLo4M8yqqcy6mDBKFkd88wTnnMhNdyVAYPME55zIkErTpev124wnOOZeRqJPBE5xzLoaicXCe4JxzMZX0EpxzLo68BOeciy1DJArkaQee4JxzGfMqqnMulgxRa+3zPJO28gTnnMtINNDXq6jOuZjyToY8270nwWdv3srSFbVIcO/UflRWFPHFb2zl0GGjpFj86M5qJpxRgZnx1W9u588zDtKlUtx/dz/GnVaR71PIWJ3Vstzms5+9AIzWWfRS3zxH1Xazkk9RTAlCiCLOLnpfvkNqs4QlWMALJEliGP0YzAidku+w0mImEpadEpykXsC9wBiiwuE/ACuB3wHHA28BV5nZLkkC7gEuAw4CN5jZwub2n9MEJ+nSEFAxcK+Z3ZnL46X66je3c8kFXfj9vQOprTUO1iT5+JTNfPOmPky+qCvTZxzglm9v57nHh/Dn5w6y+o06Vr40jLkLD/OlW7Yxe3rhPb5wlS2irwZwms4haUkS1Oc7pKw5U5MoU3yei1pEEeN4LyUqIWlJ5vM8VTaAngXyDymZvRLcPcDTZvZRSWVAF+BWYIaZ3SnpFuAW4BvAZGBkmM4GfhJ+NilnFWlJxcCPQ1Cjgasljc7V8VLt2ZvgxTk13PjJHgCUlYlePYuRYO/+ZNgmycABUX5/8ukDXPex7khi4pkV7N6bZNOWwkoO9VbHLrYxiOEAFKmIUpXlOSrXFEmUKPr+WSjFFYqok6Ekrak5knoC7wHuAzCzWjPbDVwBPBA2ewC4MsxfATxokTlAL0kDmztGLktwE4A1ZvYGgKSHQ4DLcnhMAN58u57qvsX8w1e3snjZYcadVsHd365i6h3VTL56I1+/YwfJpDHrySEAbNhcz9BB73wUQwaWsGFTPQP7F04NvoYDlFHOMnuZfbaHHvRmlMZSrMI5h+a8YjPBYLBGMEQn5DucrDAz5vJXatjPEEYUTOktw06GKknzU15PM7NpYX44sA34haTTgQXAV4D+ZrYpbLMZ6B/mBwPrUva1PizbRBNy2RXSVDA5V19vLHztMJ+/vicLnh1G10px1w938dMH9/C926tYu+B4vnd7FZ+9eWt7hNMujCT72M0QjWBi0cUUU8xbtiLfYWXFeF3I2UUXc4bOZ72tYZdty3dIWSGJibqY8/gAe9nFftuT75DSljClNQHbzWx8yjQtZTclwDjgJ2Z2BnCAqDp6hJkZtL54m/e+XklTJM2XNH/bjuw8THbIoBKGDCzh7HFRR8FHPtiNha8d5sFH9vHhD3QF4GOXd2PeK4cAGDyghHUb36mSrt9Uz+CBhVXyKacL5VQeKQX00xD2sivPUWVHhSoBKFMF1QxmLzvzHFF2laqM3lSzg835DiUtDVcypDO1YD2w3szmhtePEiW8LQ1Vz/CzoSSyAUhtHB8SljUplwkurWDMbFpDdq/um53BgwP6lTB0UAkr19QC8Nysg4w+qYxB/Yv52+yasKyGkcOjNqrLL+nKr36/DzNjzoJD9OxeVFDVU4ByVVBBFw7YPgB22la60SPPUbVdwuqpt7oj8zvZQld65jmqtqu1w9RZ9P1MWIKdbKEL3fMcVfqSVpTW1Bwz2wyskzQqLLqIqAnrSeD6sOx64Ikw/yTwKUUmAntSqrKNyuVf8cvASEnDiRLbJ4BP5vB4R7nnu9Vc96Ut1NYZw4eVcv/d/fjQJV352je3U5/YTkW5+Ol/VQNw2UVd+POMg5z092vpUlnEfVP7tVeYWTVKZ7DE5mKWpJKujNZZ+Q6pzQ5ziMX2Elj0LIABGkaVBuQ7rDY7TA1LmQ9mGEZ/hlCtQfkOKy3RxfZZKxt9Gfh16EF9A/g0UcHrEUk3AmuBq8K204mGiKwhGiby6ZZ2Lsvh03EkXQbcTTRM5H4z+25z248/vcLmPVN4wzNacsngM/IdgstUgTw1KlNzbQZ7bWebxngMP7Wbfevx09La9oaTZi8ws/FtOV5b5LQeZmbTibKucy4mzMjaQN9cK6yGJudcB6BsDvTNKU9wzrmMGF6Cc87FmN/w0jkXS4b8hpfOuXiKHhtYGKmjMKJ0znUg/uBn51xMGbR4lUJH4QnOOZcxL8E552LJTF6Cc87FU9TJ4E/Vcs7FUvaeyZBrnuCccxmJOhm8Dc45F1N+JYNzLpb8SgbnXKz5k+2dc7FkBnVJT3DOuRiKqqie4JxzMeVXMjjnYsmHiTjnYsyrqM65GPNnMrTCqsVduGTQ2HyHkXVjX4nnI+gAFvkTETudqBe1MK5FLYxypnOuw2gY6JvO1BJJb0l6TdIiSfPDsj6SnpW0OvzsHZZL0g8krZG0WNK4lvbvCc45l7FkeHRgS1OaLjCzsSkPiL4FmGFmI4EZ4TXAZGBkmKYAP2lpx57gnHMZaehFzUYJrglXAA+E+QeAK1OWP2iROUAvSQOb25EnOOdcxpJWlNYEVEmanzJNOWZXBvxF0oKUdf3NbFOY3wz0D/ODgXUp710fljWpQ3UyOOc6PjNRn/4wke0pVc/GnGdmGyT1A56VtOLoY5lJanUvnSc451zGsjXQ18w2hJ9bJf0BmABskTTQzDaFKujWsPkGYGjK24eEZU3yKqpzLiPZaoOT1FVS94Z54P3AEuBJ4Pqw2fXAE2H+SeBToTd1IrAnpSrbKC/BOecylqUSXH/gD5IgykW/MbOnJb0MPCLpRmAtcFXYfjpwGbAGOAh8uqUDeIJzzmUkWze8NLM3gNMbWb4DuKiR5QZ8KZNjeIJzzmXML9VyzsWSGdT7DS+dc3Hlt0tyzsWSP3TGORdr5gnOORdX3sngnIslM2+Dc87Flkh4L6pzLq68Dc45F0v+VC3nXHxZ1A5XCDzBOecy5r2ozrlYMu9kcM7FmVdRO6BDdpClvEwthwAxmOEM08h8h5WRaZOnU9a1BBWJopIirvvNRcz68RLWvLAJCbr0KWfyHWfRrV8l8365kuXT3wYgmTB2vrmXLz7/ISp7luX5LDJnZsxlBhVUMFbn5TucNiv072Kn70WVdD/wQWCrmY3J1XEyIcRITqOHelNvdcxjBn2sP93UI9+hZeSqn7+XLr3Lj7w+6/pRnPel6CNe+JvVzJ62nIv/bRwTbhjFhBtGAfD63zYy/6HVBZncAN5mNV3pToK6fIeSFYX8XTQrnASXy4r0L4FLc7j/jJWrkh7RM2QpUSld6M5havIcVduVdys9Ml9Xk6Cx9t/lf17HyZcOffeKAnDIDrKdTQxmeL5DyZpC/y7m+LGBWZOzEpyZzZR0fK7231Y1doB97KYnffIdSmYEj37hRSQ47SMncPpHTwDgxR8uYdmf1lLWrZSP//y9R72lrqaet17azEX/ekY+Im6zVbzKSE6jPialt2MV4nfR2+DSFJ6FOAWggi7tcsx6q2cxsxnFWEpU2vIbOpCrf3EB3ftXcmDnIR79/Iv0Gd6doWdWc/6Xx3D+l8cw974VvPLwGs794ilH3vP6zE0MGltVkNXTbbaRMsrpod7stK0tv6HAFOJ30RDJAulFzXuUZjbNzMab2fhSylt+QxslLcliZjOAYfRTs8+M7ZC6968EoGufCk68YBCbl+w8av3Jlw1j1Yyjn6S24unCrZ7uYQfb2MQsm84S5rKTbSyxefkOKysK+btoaU75lvcE157MjGXMpyvdOU4n5TucjNXW1FN7oO7I/NrZW6g6sSe71u47ss2aFzbSZ3j3I68P76tj/YJtjLhgULvHmw0n6lTO1wc4T5cxhrPpQzVjNCHfYbVZQX8XQydDOlO+5b2K2p72sIPNvE03ejLHngXgRMZQpYF5jiw9B3cc4ombZgOQrDdOnjyU4ecO4ImbZ7PzrX2oSPQY2IWL/++4I+9Z/dwGjvv7/pRVdqpfdYdX6N/FDlE8S4MsR62Fkn4LTAKqgC3AbWZ2X3Pv6aE+drbe9bSwgjf2lXxHkDuLCrPfotOaazPYazvbVLSqGDHYht75hbS2XXPVNxeY2fi2HK8tctmLenWu9u2cyx8DksnsVT8lFQPzgQ1m9kFJw4GHgb7AAuA6M6uVVA48CJwJ7AA+bmZvNbfvTtUG55zLAgNM6U3p+QqwPOX1XcBUMzsR2AXcGJbfCOwKy6eG7ZrlCc45lzGz9KaWSBoCfAC4N7wWcCHwaNjkAeDKMH9FeE1Yf1HYvkme4JxzmUt/nEiVpPkp05Rj9nQ38HUgGV73BXabWX14vR5oGEMzGFgHENbvCds3ybvWnHMZymgIyPamOhkkNVyrvkDSpGxFl8oTnHMuc9kZfHEu8CFJlwEVQA/gHqCXpJJQShsCNIxc3wAMBdZLKgF6EnU2NMmrqM65zBhYUmlNze7G7F/NbIiZHQ98AnjOzK4Bngc+Gja7HngizD8ZXhPWP2ctjHPzBOecawWlObXKN4CbJK0hamNrGD97H9A3LL8JuKWlHXkV1TmXuSxfH2BmLwAvhPk3gHddj2dmh4CPZbJfT3DOucwVyKVanuCcc5lpGOhbADzBOecy5je8dM7FVxavRc2lFntRFblW0r+H18OkGNyQyznXarL0pnxLZ5jI/wB/DzTcHWQf8OOcReSc69jSvUyrAyS4dKqoZ5vZOEmvAJjZLkmFd3N/51yWZHSnkLxKJ8HVhfs1GYCkat65MNY51xl1gNJZOtKpov4A+APQT9J3gVnAf+Q0Kudcx5ZMc8qzFktwZvZrSQuAi4iuvbjSzJa38DbnXFzFaRycpGHAQeCPqcvM7O1cBuac67g6Qg9pOtJpg3uKKGeL6JYmw4GVwCnNvck5F2NxSXBmdmrqa0njgC/mLCLnnMuSjK9kMLOFks7ORTAANH+L9YL06lnF+Q4hZ9Y9+nf5DiEnjrt2Tb5DyAkdys7fV2yqqJJuSnlZBIwDNuYsIudcx2YUzKVa6ZTguqfM1xO1yT2Wm3CccwUhDiW4MMC3u5n9czvF45wrAAVfRW146IOkc9szIOdcASj0BAfMI2pvWyTpSeD3wIGGlWb2eI5jc851VDFIcA0qiB7NdSHvjIczwBOcc51QR7kVUjqaS3D9Qg/qEt5JbA0K5PScczkRg17UYqAbjT/7yxOcc51YHEpwm8zsjnaLxDlXOAokwTV3u6TCKIM659pXmrcrb6mUJ6lC0jxJr0paKun2sHy4pLmS1kj6XcMNdiWVh9drwvrjWwq1uQR3Udon7JzrXLJzy/LDwIVmdjowFrhU0kTgLmCqmZ0I7AJuDNvfCOwKy6eG7ZrVZIIzs50thuec65SUTG9qjkX2h5elYTKiERuPhuUPAFeG+SvCa8L6i6TmL15P546+zjnXWlWS5qdMU1JXSiqWtAjYCjwLvA7sNrP6sMl6YHCYHwysAwjr9wB9mzu4PxfVOZe59DsZtpvZ+CZ3Y5YAxkrqRfRohKzensZLcM65zGSpk+GoXZrtBp4nekRpL0kNha8hwIYwvwEYCtGlpEBPoosQmuQJzjmXuSx0MkiqDiU3JFUCFwPLiRLdR8Nm1wNPhPknw2vC+ufMrNmjeBXVOZe57IyDGwg8EO5aVAQ8YmZ/krQMeFjSd4BXgPvC9vcBv5K0BtgJfKKlA3iCc85lRLTcQ5oOM1sMnNHI8jeACY0sPwR8LJNjeIJzzmUmJhfbO+dc4zzBOediyxOccy6uvIraQc1KPkUxJQghiji76H35DqnVltbPZZttpIwKzimdDMCW5Nu8nljCAfYyofj99Czqk+co02eJJGu/MY2SPt0Zcus1bLz7MQ69vhEVF1ExcjADPnc5Kinm4JI32XDXw5T26wVAt7NPpuqqSfkNPk1LamezLbGeMlVwbsXlAOxN7mR57TySJBDi5LIJ9CyqynOkLejsCU7SUOBBoD/RxzHNzO7J1fEycaYmUabyfIfRZoOKhjNUI1lSP/fIsq7qyekl57E88XIeI2udXU/NoWxwFcmawwD0OP9UBn7lwwBsmvoYu/+6kN6XngVA5cnDGHLrNXmLtbUGFZ/AsJKTeK32pSPLVte9wgmlp1JdPJhtiQ2sqlvIWeXvz2OULbDs9KK2h1wO9K0Hbjaz0cBE4EuSRufweJ1O76J+lFJ21LJu6klX9chTRK1Xt2MPBxaupuf7xh1Z1u3Mk5CEJCpGDqZ+x948RpgdfYr7U8q7/7kmrA6AequlXF3aO6zMZeduIjmXsxKcmW0CNoX5fZKWE10suyxXx0zXKzYTDAZrBEN0Qr7DccDW+5+m+rqLj5TeUll9gr1/e5V+/zD5yLKalet566afUNKnO9Wfej/lw/q1Z7hZNap0PAtrZ7CyfiGYMaH8knyH1CJvg0sRbkx3BjC3kXVTgCkAFeT+P9d4XUiFKqm1Qyy0mXSlO71VnfPjuqbtn7+Skp5dqRgxiINL3nzX+i0/f4ouo4+jy+jjACg/YSAjfvpViirL2b9gFRvuepgTfvxP7R121qyvX8Wo0vH0Lx7G5vq1LK2bw/jyDt42XCAJLufXokrqBjwGfNXM3lXHMLNpZjbezMY3VnTPtgpVAlCmCqoZzF78tnf5VrNiHftfXsnrn5/KxqmPcvC1N9l4z2MAbH/kBRJ7DlB9wzulmuIuFRRVRt+VbmeehCUS1O890Oi+C8HGxBv0KxoKQP/iYexJNnv9eP6lWz3tAEkwpyU4SaVEye3XHeE5qgmrxzBKVErC6tnJFobjzYL5Vn3t+6i+NiqxHFzyJjuffIlBX/kIu/+6gAOL1jD0tutR0Tv/i+t37aO4VzckUbN6PZhR3L0A2q2aUK5KdiW30Kd4ADuTm+mi7vkOqVnCq6iEO23eByw3s+/n6jiZOMwhFttLYGAYAzSMKg3Id1ittrj+JXbZVuo4zMy6JxhRPIZSylmRWEAth1mU+Bvdk70ZVzIp36G2ypaf/YnS6l68feu9wDvDQfbNXsbuZ+aj4iJUVsKgr32UFm7s2mEsrn2RnYkt1HGYv9U8zojS0xhdOpEVdfOxuiRFFHNK2dn5DrNFhZLg1MLdRlq/Y+k84EXgNaChU/lWM5ve1Ht6qI8V8ri0pqi4ON8h5MzbD2f1/oQdxnHXrsl3CDkx59B09iR3tOm/QZf+Q23kJ25Ka9vFP7hpQXM3vMy1XPaizsKfzOVcPBVICa7TXcngnGsjv5uIcy7WPME55+KqUC7V8gTnnMuYV1Gdc/HUQQbxpsMTnHMuc57gnHNx5FcyOOdiTcnCyHCe4JxzmSmgNjh/sr1zLmOy9KZm9yENlfS8pGWSlkr6SljeR9KzklaHn73Dckn6gaQ1khZLGtf8ETzBOedaIzu3S2rqrt+3ADPMbCQwI7wGmAyMDNMU4CctHcATnHMuY9kowZnZJjNbGOb3AQ13/b4CeCBs9gBwZZi/AnjQInOAXpIGNncMT3DOucylX4KrkjQ/ZZrS2O6Ouet3/z4cvokAAAmrSURBVPDIA4DNRA+ugij5rUt52/qwrEneyeCcy0xmT9Xa3tLtko6963fqvf3MzKTWD0rxEpxzLiMN4+DaWkWFJu/6vaWh6hl+bg3LNwBDU94+JCxrkic451zmzNKbmtHMXb+fBK4P89cDT6Qs/1ToTZ0I7EmpyjbKq6jOuYxl6UqGc4HrgNckLQrLbgXuBB6RdCOwFrgqrJsOXAasAQ4Cn27pAJ7gnHOZydJA3xbu+n1RI9sb8KVMjuEJzjmXMb8fnHMutjzBOefiyWixA6Gj8ATXDiyRyHcIORPXx+utuf2MfIeQE4fueSEr+/HbJTnn4ssTnHMujvyGl865+DLzG14652KsMPKbJzjnXOa8iuqciycDvIrqnIutwshvnuCcc5nzKqpzLra8F9U5F08F9NhAT3DOuYxEA30LI8N5gnPOZc7vJuKciysvwTnn4snb4Jxz8eXXojrn4syrqM65WMrswc955QnOOZc5L8E552KrMPKbP9neOZc5JZNpTS3uR7pf0lZJS1KW9ZH0rKTV4WfvsFySfiBpjaTFksa1tH9PcM65zBjRQN90ppb9Erj0mGW3ADPMbCQwI7wGmAyMDNMU4Cct7dwTnHMuI8KQpTe1xMxmAjuPWXwF8ECYfwC4MmX5gxaZA/SSNLC5/Xe6NrhZyacopgQhRBFnF70v3yFlTVzObUntbLYl1lOmCs6tuByAvcmdLK+dR5IEQpxcNoGeRVV5jrRlybo6Nv30x1h9PZZM0vXU0+jz/kvZ+puHOLx+PRQXUz50KNUf+RgqLmb3C8+z/5WFAFgySd3WLRx32x0Ud+mS5zM5RvqdDFWS5qe8nmZm01p4T38z2xTmNwP9w/xgYF3KduvDsk00IWcJTlIFMBMoD8d51Mxuy9XxMnGmJlGm8nyHkRNxOLdBxScwrOQkXqt96ciy1XWvcELpqVQXD2ZbYgOr6hZyVvn78xhlelRSwsApX6CovBxLJNj4Pz/i0KiT6XbGmVRffQ0AW3/zEPvmzaXH359Dr0kX0GvSBQAcWLaUPS/O7HjJDTJJcNvNbHzrD2Mmtf7uc7msoh4GLjSz04GxwKWSJubweC4m+hT3p5R3J+mE1QFQb7WUqwP+0TdCEkXl0blYIhE9BFzQ5eSTkYQkyocOo37P7ne9d/+iV+g2tgM+gDq7bXCN2dJQ9Qw/t4blG4ChKdsNCcualLMSnJkZsD+8LA1Th+hcfsVmgsFgjWCITsh3OFkV13MbVTqehbUzWFm/EMyYUH5JvkNKmyWTbLhnKnU7ttPjnHOpGHbcO+sSCfYvXEDfK6486j3J2lpqVq6g6soPt3e4aUmnh7QNngSuB+4MP59IWf6Pkh4Gzgb2pFRlG5XTNjhJxcAC4ETgx2Y2N5fHS8d4XUiFKqm1Qyy0mXSlO71Vne+wsiLO57a+fhWjSsfTv3gYm+vXsrRuDuPLC6ONUUVFDPnazSRqatjywC+o3byJsgFR2/j2PzxGxQknUDn86H9GB5ctpeL44R2zeoplbaCvpN8Ck4ja6tYDtxEltkck3QisBa4Km08HLgPWAAeBT7e0/5wmODNLAGMl9QL+IGmMmS1J3UbSFKIuXyrI/S+zQpUAlKmCahvMXnbSm3gkgTif28bEG4wqjZpy+hcPY2ndnDxHlLniykoqR5zIwZUrKBswkF3PPkPiwH76f/iGd227/9VFHbN6CuFuItlJcGZ2dROrLmpkWwO+lMn+22WYiJntBp7n3eNdMLNpZjbezMY31u6STQmrpz604ySsnp1soSs9c3rM9hLncwMoVyW7klsA2JncTBd1z3NE6Uns30+ipgaIelRrVq+itLo/e+fO4eCqlfT75HWo6Og/w2RNDYfeeJ0up5ySj5DTk9s2uKzJZS9qNVBnZrslVQIXA3fl6njpOMwhFttLUQkbY4CGUaUB+Qwpa+J0botrX2RnYgt1HOZvNY8zovQ0RpdOZEXdfKwuSRHFnFJ2dr7DTEv9vr1s+91vIWmYGd1OO52uo0fzxi3/Qkmv3mz80Q8A6DrmVHpfHPUKH1j6GpUnjaKorOP2hhfKDS9lOQpU0mlEg/SKiUqKj5jZHc29p4f6WKGO3eqsGnoI42bN7R20ethG6++ZyuH169SWffSsHGjnHH9DWts+veLOBW0ZJtJWuexFXQzE81viXGdmBokOUP9MQ6e7ksE5lwUFUkX1BOecy5wnOOdcLBngz2RwzsWTgXkbnHMujgzvZHDOxZi3wTnnYssTnHMunrJ3sX2ueYJzzmXGgNzeLilrPME55zLnJTjnXDz5pVrOubgyMB8H55yLLb+SwTkXW94G55yLJTPvRXXOxZiX4Jxz8WTR810LgCc451xm/HZJzrlY82Eizrk4MsC8BOeciyXzG14652KsUDoZcvZc1NaQtA1Y206HqwK2t9Ox2pOfV+Fpz3M7zsyq27IDSU8TxZyO7WZ2aVuO1xYdKsG1J0nz8/lA2lzx8yo8cT63fCvKdwDOOZcrnuCcc7HVmRPctHwHkCN+XoUnzueWV522Da4zkZQAXiPqNV8OXG9mB1u5r18CfzKzRyXdC3zfzJY1se0koNbMXsrwGG8B480srp0Krp105hJcZ1JjZmPNbAxQC3w+daWkVg0XMrPPNJXcgknAOa3Zt3PZ4Amu83kROFHSJEkvSnoSWCapWNJ/SXpZ0mJJnwNQ5EeSVkr6K9CvYUeSXpA0PsxfKmmhpFclzZB0PFEi/ZqkRZLOl1Qt6bFwjJclnRve21fSXyQtDaVCte9H4uLKB/p2IqGkNhl4OiwaB4wxszclTQH2mNlZksqB/5X0F+AMYBQwGugPLAPuP2a/1cDPgfeEffUxs52SfgrsN7P/Dtv9BphqZrMkDQOeAU4GbgNmmdkdkj4A3JjTD8J1Gp7gOodKSYvC/IvAfURVx3lm9mZY/n7gNEkfDa97AiOB9wC/NbMEsFHSc43sfyIws2FfZraziTjeB4yWjhTQekjqFo7x4fDepyTtauV5OncUT3CdQ42ZjU1dEJLMgdRFwJfN7Jljtrssi3EUARPN7FAjsTiXdd4G5xo8A3xBUimApJMkdQVmAh8PbXQDgQsaee8c4D2Shof39gnL9wHdU7b7C/DlhheSGpLuTOCTYdlkoHfWzsp1ap7gXIN7idrXFkpaAvyMqIT/B2B1WPcgMPvYN5rZNmAK8LikV4HfhVV/BP5PQycD8E/A+NCJsYx3enNvJ0qQS4mqqm/n6BxdJ+Pj4JxzseUlOOdcbHmCc87Flic451xseYJzzsWWJzjnXGx5gnPOxZYnOOdcbP1//wxY1t4kppsAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "19RtF4qEuIjy"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GgimVF2ouIjy",
        "outputId": "144e0479-128a-492c-8b44-c3760541d2f1"
      },
      "source": [
        "# Salvando o modelo para posterior reuso\n",
        "dump(grid_search.best_estimator_, 'modelo_tf-idf_sklearn_tcu_jurisp.pkl')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['modelo_tf-idf_sklearn_tcu_jurisp.pkl']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3Ojj9ATjuIjz"
      },
      "source": [
        "# Carregar o modelo do disco\n",
        "loaded_model = load('modelo_tf-idf_sklearn_tcu_jurisp.pkl')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x68dew5QuIj0",
        "outputId": "e4bb3358-1031-4aaa-ac1b-985216ec308b"
      },
      "source": [
        "# usar o modelo carregado para inferência.\n",
        "result = loaded_model.score(text_test, target_test)\n",
        "result"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9590842669264491"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OGTaZRu7uIj0"
      },
      "source": [
        "### FIM"
      ]
    }
  ]
}