grand-debat.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><h2>Détection de thème sur une question environnement du grand débat</h2></center>\n",
    "\n",
    "Fork du projet TALAGRAND (https://github.com/Quantmetry/grand-debat) qui met à disposition des techniques d'IA pour appréhender le nombre important de réponses au grand débat.  \n",
    "\n",
    "De même mon but est d'essayer des techniques d'extraction de thème (topic detection) sur des questions du grand débat. Pour cet essai j'ai pris : \"de quelle manière votre vie quotidienne est-elle touchée par le changement climatique ?\").  \n",
    "J'ai tenté de le comparer ensuite à la synthèse \"officielle\" réalisée par OpinionWay : \n",
    "https://granddebat.fr/media/default/0001/01/b88758e8caa2733bec607a74b3b5371cc0a3b420.pdf\n",
    "\n",
    "\n",
    "Tel quel, le projet de départ TALAGRAND m'a donné des thèmes difficilement interprétables. Dans cette implémentation j'ai modifié la préparation de données grâce à la librairie  **spacy (lemmatisation, filtrage par POS)**, utilisé le même algo de **LDA de scikit-learn** que le projet initial, et déterminé le nombre de thèmes grâce au **score de cohérence** fourni par la librairie **gensim**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%config Completer.use_jedi = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings; warnings.simplefilter('ignore')\n",
    "import logging\n",
    "logging.getLogger().setLevel(logging.ERROR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Téléchargement des données\n",
    "-  Récupérer la liste des thèmes en analysant le site du Grand Débat\n",
    "-  Sélectionner un thème en modifiant la variable `selected_theme`\n",
    "-  Télécharger ensuite le fichier json le plus récent associé à un thème depuis datagouv.fr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#from grand_debat.downloading import download_data, get_themes\n",
    "\n",
    "#themes = get_themes() # not working\n",
    "\n",
    "themes = {\"1 L'organisation de l'État et des services publics\": \n",
    "              \"http://opendata.auth-6f31f706db6f4a24b55f42a6a79c5086.storage.sbg.cloud.ovh.net/\"+\n",
    "              \"2019-03-21/ORGANISATION_DE_LETAT_ET_DES_SERVICES_PUBLICS.json\",\n",
    "          \"2 la-transition-ecologique\": \n",
    "              \"http://opendata.auth-6f31f706db6f4a24b55f42a6a79c5086.storage.sbg.cloud.ovh.net/2019-03-21/\"+\n",
    "              \"LA_TRANSITION_ECOLOGIQUE.json\",\n",
    "          # 2 autres thèmes (fiscalité, démocratie) à voir ici:\n",
    "          # https://granddebat.fr/pages/donnees-ouvertes\n",
    "         }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choisir le numéro associé au thème que l'on souhaite étudier\n",
    "selected_theme = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#download_data(themes, selected_theme)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chargement des données\n",
    "-  Charger le fichier json précédemment téléchargé\n",
    "-  Afficher les différentes questions associées au thème sélectionné\n",
    "-  Sélectionner une question à étudier (variable `selected_question`)\n",
    "-  Afficher le nombre de réponses existantes correspondant à cette question"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from grand_debat.loading import load_answers, get_path, get_questions_from_json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = get_path(themes, selected_theme)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions = get_questions_from_json(path)\n",
    "#questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Si oui, de quelle manière votre vie quotidienne est-elle touchée par le changement climatique ?'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Indiquer ici l'identifiant associé à la question étudiée\n",
    "selected_question = 'UXVlc3Rpb246MTQ3' # Si oui, de quelle manière votre vie quotidienne est-elle touchée ...\n",
    "questions[selected_question]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "153809it [00:07, 20121.12it/s]\n"
     ]
    }
   ],
   "source": [
    "answers = load_answers(path, selected_question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nombre de réponses analysées : 92795\n"
     ]
    }
   ],
   "source": [
    "print(\"Nombre de réponses analysées :\", len(answers))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Détection de thèmes</h2>\n",
    "\n",
    "-  Détecter les thèmes principaux associés à la question sélectionnée en utilisant l'algorithme **LDA** (Latent Dirichlet Allocation)\n",
    "-  Le resultat de l'analyse se trouve dans le fichier pyLDAVIS_tf.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/gensim/matutils.py:22: DeprecationWarning: Please use `triu` from the `scipy.linalg` namespace, the `scipy.linalg.special_matrices` namespace is deprecated.\n",
      "  from scipy.linalg.special_matrices import triu\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<module 'grand_debat.theme_detection' from '/Users/charlesprat/RepoGit/grand-debat/grand_debat/theme_detection.py'>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "import importlib\n",
    "import grand_debat.theme_detection as td\n",
    "importlib.reload(td)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparation des données\n",
    "\n",
    "Tokenisation, lemmatisation, filtrage sur type (POS NOUN et ADJ) et stop words de spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<module 'grand_debat.theme_detection' from '/Users/charlesprat/RepoGit/grand-debat/grand_debat/theme_detection.py'>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "importlib.reload(td)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "load spacy pipeline\n",
      "tokenizing datas\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 92795/92795 [02:09<00:00, 714.04it/s]\n",
      "100%|██████████| 1/1 [00:00<00:00, 324.56it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Additional stopwords  {'oui', 'climatique', 'vie', 'changement', 'manière', 'quotidien', 'est-ell', 'toucher'}\n",
      "fitting countvectorizer\n",
      "data preparation done\n"
     ]
    }
   ],
   "source": [
    "gd_prep = td.GDebatDataPreparation(answers, stopsentence=questions[selected_question])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "4970"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(gd_prep.tf_bow.vocabulary_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recherche du nombre optimal de thèmes pour l'entrée de la LDA\n",
    "\n",
    "On se base sur le \"coherence score\".  \n",
    "https://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<module 'grand_debat.theme_detection' from '/Users/charlesprat/RepoGit/grand-debat/grand_debat/theme_detection.py'>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "importlib.reload(td)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "logging.getLogger().setLevel(logging.ERROR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "from numpy.random import default_rng\n",
    "rng = default_rng(seed=11)\n",
    "\n",
    "num_topics = [4,5,6,7,8,9,10,12,14,17,20]\n",
    "answs_samples = rng.choice(gd_prep.answ_lems, size=20000, replace=False).tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "compute lda for 4 topics\n",
      "Lda done, coherence 0.609\n",
      "compute lda for 5 topics\n",
      "Lda done, coherence 0.670\n",
      "compute lda for 6 topics\n",
      "Lda done, coherence 0.714\n",
      "compute lda for 7 topics\n",
      "Lda done, coherence 0.690\n",
      "compute lda for 8 topics\n",
      "Lda done, coherence 0.716\n",
      "compute lda for 9 topics\n",
      "Lda done, coherence 0.688\n",
      "compute lda for 10 topics\n",
      "Lda done, coherence 0.634\n",
      "compute lda for 12 topics\n",
      "Lda done, coherence 0.660\n",
      "compute lda for 14 topics\n",
      "Lda done, coherence 0.665\n",
      "compute lda for 17 topics\n",
      "Lda done, coherence 0.573\n",
      "compute lda for 20 topics\n",
      "Lda done, coherence 0.551\n",
      "Best coherence achieved for n_topics = 6\n"
     ]
    }
   ],
   "source": [
    "v_coh, l_topics, best_n = td.compute_coherence_vs_ntopics(\n",
    "    answs_samples, gd_prep, num_topics=num_topics, lambda_coh=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['hiver', 'été', 'chaud', 'degré', 'doux']\n",
      "['disparition', 'oiseau', 'insecte', 'espèce', 'animal']\n",
      "['agricole', 'légume', 'bio', 'habitude', 'action']\n",
      "['inondation', 'littoral', 'glacier', 'érosion', 'fonte']\n",
      "['pollution', 'air', 'allergie', 'respiratoire', 'asthme']\n",
      "['voiture', 'transport', 'énergie', 'chauffage', 'taxe']\n",
      "['enfant', 'avenir', 'futur', 'peur', 'inquiétude']\n",
      "['température', 'hausse', 'tigre', 'écart', 'gel']\n",
      "['violent', 'phénomène', 'épisode', 'vent', 'météorologique']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "for topic in l_topics[5]:\n",
    "    print(topic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA1X0lEQVR4nO3deXhU5dnH8e+dnZAQCCFhSdgTwioQZFcIKOKOS622dalatC22LqVi+9a2tu/bVlu1dbeubVW0KopURYSgLIKALLIFkrAlLCEBYgJkv98/ZrBjSMgkmeRMMvfnuuZi5mzzO8Pk3HOeszyiqhhjjAk8QU4HMMYY4wwrAMYYE6CsABhjTICyAmCMMQHKCoAxxgSoEKcDNERcXJz27t27UfMeP36c9u3b+zaQD1iuhrFcDWO5Gqat5lq3bl2BqnY5bYSqtppHWlqaNlZGRkaj521OlqthLFfDWK6Gaau5gLVayzbVmoCMMSZAWQEwxpgAZQXAGGMClBUAY4wJUFYAjDEmQFkBMMaYAGUFwBhjApQVAHOa7MMlfHm40ukYxphmZgXAfENpRRU3vfg5j3xRRlZ+idNxjDHNyAqA+YYnl2az78hJQgT+8P42p+MYY5qRFQAHfbrjMJ/sq3A6xtd2Fxzn6U+yuXx4dy7vH8ri7fmsyCpwOpYxpplYAXCIqvKb+Vt4cUs5q3IKnY7jyvPeFsKCg/jFRQM5v1coPTq24/f/2UZVtXUbakxbZAXAIRtzi8gpOE6wwM/f3MSJcmcPun609RBLMw9z53nJJHSIICxYuPfCVLYd+Iq31uU6ms0Y0zysADjknfV5hIUEMWtEOHuPnODBDzMdy3KyvIoH3ttKatdobhrf++vhlw7rxoieHfnzR5kcL7Ozgoxpa6wAOKCiqpr3Nu7n/IEJjIgP4abxvXlp5W5WO9QU9HjGTvKOneSBy4cQEvzfr4SI8D8XDyK/uIxnPs1xJJsxpvlYAXDAsp2HKTxezowRPQD4+fQB9IyN5OdvbeJkeVWLZsk5XMKzn+Zw5YgejO4Te9r4tF6duHhYN579NJsDRSdbNJsxpnlZAXDA21/k0SkylEkprg56IsNC+NNVw9hTeIKHFrZcU5Cq8uv5W4gICea+iwbWOd2c6alUV8OfF+5osWzGmOZnBaCFFZdWsGjrIS4Z1p2wkP9+/OP6deaGcb14ceUu1uw+0iJZPth8kGU7C7hnWgpdosPrnC4pNpLvT+jNW1/ksjmvqEWyGWOanxWAFvbB5oOUVVZzxcgep427d3oqPTq24+dvNn9T0PGySn63YCuDunXge2N71Tv9j9L7E9s+jN//ZyuuHuaMMa2dFYAW9s76PHp3jmREUsfTxrUPD+HBq4axq+A4Dy9q3qagx5ZkcaColN/NGPyNA791iWkXyl3nJbMq5wiLth5q1mzGmJZhBaAFHSg6yWc5hcwY0QMRqXWa8f3j+N7Ynjy3fBfr9jRPU1BWfjHPLcvhW2mJpPU6/cBvXa4b3ZN+Xdrzhw+2U15Z3SzZjDEtxwpAC3p3w35UYcbw05t/PM25cCDdY9ox+81NlFb4tilIVbn/3S1EhgUz58LUBs0bEhzELy8eyK6C47yyeo9PcxljWp4VgBaiqsz7Io+RPTvSO679GaeNCnedFZRz+DiPLPLtmTfvbTrAyuxCZk9PpXNU3Qd+65I+IJ4J/Tvz18U7KTrhP/cxMsY0nBWAFrLtQDGZh4q5YsSZf/2fMjE5jutG9+Tvy3L4Yu9Rn2QoKavk9wu2MrRHDN8Z3bNRyxARfnnRIIpOVvDYkp0+yWWMcYYVgBYyb30uIUHCxcO6ez3PLy5KpWuHCGb/e6NPmoL++vEODpeU8bsZQwgOqv0YhDcGde/At9ISefmz3ewuON7kXMYYZ3hVAERkuohkikiWiMypZfwjIrLB/dghIsc8xt0oIjvdjxs9hqeJyJfuZf5N6joq2gZUVSvvbtjP5AHxxLYP83q+6IhQ/njVMLIPH+fRj5v2azvzYDEvrNjNtWcnMbyWM5Aa6p5pAwgNDuJPH25v8rKMMc6otwCISDDwBHAhMAi4TkQGeU6jqnep6nBVHQ48BrztnjcW+DUwBhgN/FpEOrlnewr4AZDsfkz3xQr5o5XZBeQXl3nd/OPp3JQuXHt2Es9+ms2Gfcca9f6qyq/e3Ux0RAg/v6BhB37rktAhgtsn9eODzQf5fFfLXLhmjPEtb/YARgNZqpqjquXAXODyM0x/HfCa+/kFwCJVPaKqR4FFwHQR6QZ0UNVV6rqq6B/AjMauhL+btz6P6PAQpg6Mb9T8v7h4IAlNaAp6Z0Men+86wr3TU+nUgD2Q+vzgnL507RDB7/+zlWrrM8CYVkfqu6pTRK4Gpqvqre7X1wNjVHVWLdP2AlYBiapaJSI/AyJU9ffu8b8CTgJLgT+q6nnu4ecA96rqJbUscyYwEyAhISFt7ty5jVrRkpISoqKiGjVvU5RVKj/JOMGYbiHcPOT0s268zbXpcCUPryvjkr6hXJ3i/Ub8RIUyZ9lJ4toJ/zM2giAvW9q8zbUir4K/f1nOzGHhjO8e4nWuxnLq/7E+lqthLFfDNDVXenr6OlUdVXO4r/9irwXeVFWfnbyuqs8CzwKMGjVKJ0+e3KjlLF26lMbO2xTvbsijrGoDP7poFGP7dm50rslArmzkrS/ymHnRaIYldvTq/X/73haKK3bzyswJDE2M8Tq3t7nOrVY+O7KcBXvKuftb5xARGuz1ezSGU/+P9bFcDWO5Gqa5cnnTBJQHJHm8TnQPq821/Lf550zz5rmfe7PMVu3tL/Lo0bEdo3t7f8VtXX558SC6RIUz+9+bKKusv8Zu3f8VL6/czXfH9GzQxr8hgoJcp4XuLyrl+eW7muU9jDHNw5sCsAZIFpE+IhKGayM/v+ZEIpIKdAI+8xi8EJgmIp3cB3+nAQtV9QDwlYiMdZ/9cwPwbhPXxe8cLi5j2c7DXD68O0FNOO3ylJh2ofzhyqFkHirm8SVZZ5y2ulq5/93NdIwMY/Y03xz4rcu4fp2ZNiiBJzOyyC8ubdb3Msb4Tr0FQFUrgVm4NubbgDdUdYuIPCAil3lMei0wVz0OKqjqEeB3uIrIGuAB9zCAHwHPAVlANvCBD9bHr7y3cT/VSqPO/qlLemo8V41M5Mml2We8NfPb6/NYu+cocy5MJSYy1GfvX5c5F6ZSVlnNI4vs4jBjWguvjgGo6vvA+zWG3V/j9W/qmPcF4IVahq8FhngbtDWatz6PIT06kJwQ7dPl3n/JIJbtPMzP/r2R+bMmfqNfAYCiExX84f1tjOzZkatHJtaxFN/q2yWK68f14uWVu7lxfC9Su3Zokfc1xjSeXQncTLLyi/kyr6jeG781RkxkKP93xVC2Hyzm8YzTm4L+siiToyfK+d2MIT5pevLWT6cmExUewv/+Z1uLvacxpvGsADSTeevzCBK4bLj3t35oiPMGJXDliB48mZH1jaagzXlF/GvVHm4Y15vB3ZvnwG9dOkaG8ZOpySzbWcDSzPwWfW9jTMNZAWgG1dXKO+v3MzG5C/HREc32PvdfOohO7cOY/eYmyiurqa5W/uedzcS2D+eu81Oa7X3P5IZxvendOZL//c82KquszwBj/JkVgGawZvcR8o6d5EofHvytTcfIMP7viqFsO/AVTy7N4t/r9rFh3zF+cVEqMe2a/8BvbcJCgphzYSo780t4fe0+RzIYY7zT/JduBqB3NuQRGRbMtMEJzf5e5w9KYMbw7jy+JIv24SGM7h3r07OOGuOCwV0Z3TuWhz/awWVndSc6wpliZIw5M9sD8LHSiioWbDrA9MFdiQxrmfr660sH0zEyjJKySh6YMbjO7iZbiojwP5cMpPB4OU8tzXY0izGmbrYH4GMZ2/MpLq1kRgv+Cu/UPox/3DyaA0Un/eb0y2GJHbliRA+eW76L74zpSWKnSKcjGWNqsD0AH3t7fR5dosOZ0D+uRd93UPcOTB3Y/E1ODTH7ggEI8NDCTKejGGNqYQXAh44eL2dpZj6Xn9W9ST1utRXdO7bje2N78Z9NB/iq1PoPNsbfWAHwoQVfHqCiSrlipLMHYf3JhUO6UlmtLNtR4HQUY0wNVgB86J31eaQkRDGom3+0w/uDET070TEylMXbDzkdxRhTgxUAH9lTeJx1e45yxYhEx8/C8SfBQcLklC4szTxMlfUaZoxfsQLgI++s348IXN5Mt35ozaYMTODI8fJG92lsjGkeVgB8QFWZtz6XsX06071jO6fj+J1JyV0IDhIyttv9gYzxJ1YAfGDDvmPsLjzh+BW4/iomMpRRvTqx2AqAMX7FCoAPzFufR3hIENOHdnU6it+aOjCebQe+Yv+xk05HMca4WQFoooqqat7buJ/zBiXQwe55U6cpqa6L1JbYXoAxfsMKQBN9knmYoycqmv3On61dvy7t6RkbaQXAGD9iBaCJ5m3II7Z9GOemdHE6il8TEaakxrMiq4CT5VVOxzHGYAWgSb4qrWDR1kNcOqwbocH2UdZn6sB4yiqr+SzHrgo2xh/YVqsJPvzyIOWV1S1658/WbHSfWNqHBbN4mzUDGeMPvCoAIjJdRDJFJEtE5tQxzTUislVEtojIq+5h6SKyweNRKiIz3ONeEpFdHuOG+2qlWsrb63PpE9ee4UkdnY7SKoSHBHNOcheWbM9H1a4KNsZp9RYAEQkGngAuBAYB14nIoBrTJAP3ARNUdTBwJ4CqZqjqcFUdDkwBTgAfecw6+9R4Vd3Q9NVpOXnHTrIq5wgzhvewWz80wJTUeA4UlbLtQLHTUYwJeN7sAYwGslQ1R1XLgbnA5TWm+QHwhKoeBVDV2vbxrwY+UNUTTQnsL97dkAdgF3810ORU18HyJXZzOGMcJ/XtiovI1cB0Vb3V/fp6YIyqzvKY5h1gBzABCAZ+o6of1ljOEuBhVV3gfv0SMA4oAxYDc1S1rJb3nwnMBEhISEibO3duo1a0pKSEqKioRs1bk6ryyxUniQwR/mds02794MtcvtScuX772UmChUZ9doH4eTWF5WqYtporPT19naqOOm2Eqp7xgeuX+3Mer68HHq8xzQJgHhAK9AH2AR09xncDDgOhNYYJEA68DNxfX5a0tDRtrIyMjEbPW9OXuce0170L9J+f7W7ysnyZy5eaM9eji3Zo7zkLtKC4tMHzBuLn1RSWq2Haai5grdayTfWmCSgPSPJ4nege5ikXmK+qFaq6C9feQLLH+GuAear6dbdQqnrAna0MeBFXU1Or8M76PEKDhYuHdnM6Sqs0dWA8qrA087DTUYwJaN4UgDVAsoj0EZEw4Fpgfo1p3gEmA4hIHJAC5HiMvw54zXMGEenm/leAGcDmBqd3QGVVNe9u3E/6gHg6tQ9zOk6rNLh7BxI6hNtVwcY4LKS+CVS1UkRmAQtxte+/oKpbROQBXLsV893jponIVqAK19k9hQAi0hvXHsQnNRb9ioh0wdUMtAG43Ter1LxWZhdyuLjMDv42gYiQPiCe/2w6QEVVtV1EZ4xD6i0AAKr6PvB+jWH3ezxX4G73o+a8u4HTtpaqOqWBWf3Cuxv20yEihPTUeKejtGpTUuOZu2Yfa3YfYXy/OKfjGBOQ7KdXA6gqn+48zKQB8USEBjsdp1Wb0D+OsJAglthVwcY4xgpAA2Tll3C4uIwJ/To7HaXVax8ewri+ne04gDEOsgLQACuzCwHXr1fTdFMHxpNTcJycwyVORzEmIFkBaIAVWQUkxbYjKTbS6ShtQvoA13EU2wswxhlWALxUVa2syilkfF/79e8rSbGRpCREWQEwxiFenQVkYHNeEV+VVjK+v7X/+9KU1ASeW5ZDcWkF0dalZpMcPV7OjkPF7MgvYeehYnYcKqa46CRjJ1TZSQumVlYAvLQi29WJiZ2y6FtTB8bz9CfZLNtZwEV2ZbVXik5WuDfwJa4Nvvt5Qcl/b6UVFR5C77hIthRW8+rqvdw8sY+DiY2/sgLgpc+yCxmQEE2X6HCno7QpI5I60jEylMXb8q0A1FBcWsGOQyVfb+x35rs29oe++u+GPjIsmOT4KCYP6EJKQhTJCdEMSIimW0wEIsKFD37Ak0uz+PbZSbQPtz938032jfBCWWUVa3Yf4brRPZ2O0uaEBAcxOaULSzPzqapWgoMCr2+F42WV7Mx3/Zr/emN/qJj9RaVfTxMRGkT/+Cgm9I8jJSHatbGPj6ZHx3YEneEzuyo5jN+vLuWllbv5cXr/llgd04pYAfDCF3uOUVpRzQRr/mkW6anxvLNhPxtzjzGyZyen4zSbk+VVZOV7Ntu4NvZ5x05+PU1YSBD9u0Qxuk8syQnRX2/sEztFNqo49u8UzJTUeJ75JJvvje1FTDs7zmL+ywqAF1ZmFxAkMLpvrNNR2qRJKV0IDhIytue3iQJQWuHa0LuabP7bhLPv6AlOdb8RFhxE3y7tGdmrE9eeneTe2EfRq3N7n+8F3TMthYv/tpy/f5rDzy4Y4NNlm9bNCoAXVmQVMCyxIx3sLJVm0TEyjLRenVi8LZ97prW+DdTuguO8tbOcV/euZWd+CXsKj1Pt3tCHBAl94toztEcMV41M/LqdvnfnSEJa6CZ4g7vHcPHQbrywYhc3TehNXJQdxzIuVgDqUVJWycbcIm6f1NfpKG3a1NR4/vDBdg4UnaRbTNN6WWtJH205yD1vbOR4eSV94kpI7RrNpWd1JyUhipSEaHp3bk9YiPOX29x1fgofbD7AU0uz+dUlg+qfwQQEKwD1+HxXIVXVau3/zWzqQFcBWLI9n++O6eV0nHpVVSuPfryDx5ZkMSwxhhv7lXPVhZOdjlWn/vFRXDkykX+u2sOt5/RpVUXWNB/nf5r4uRVZhYSFBDGyV+tvm/Zn/bpEkRTbrlXcHbToRAW3vLyGx5Zk8a20RN64bRyd2/n/n9JPpyajqjy2JMvpKMZP+P+31mErsgoY1auTXUnZzESEqakJrMguoLSiyuk4ddp24CsufXw5K7IK+P2MITx49bBW891Iio3k2rN78saafewtPOF0HOMHrACcQUFJGdsPFtvdP1vIlNR4Siuq+cx911V/8+6GPK54cgVllVXMnTmO743thatH09Zj1pT+BAcJj368w+koxg9YATiDVTmuDdF4u/9/ixjTN5bIsGAWbz/kdJRvqKiq5ncLtvLTuRsY2iOG9+6YSForbRJM6BDBjeN7M29DHjsPFTsdxzjMCsAZrMgqJDo8hKE9YpyOEhDCQ4I5JzmOJdvy0VMnzDusoKSM7z23mueX7+Km8b155daxxEdHOB2rSW6f1I/2YSE8vMj2AgKdFYAzWJldwJi+nVvsfG0DU1MT2F9UyvaDzv863bDvGJc+tpwN+47x8DVn8ZvLBvvFKZ1NFds+jJsn9uGDzQf5MrfI6TjGQa3/29xMco+eYE/hCSbY7Z9b1OTULoDzncTM/Xwv1zz9GcFBwls/HM+VIxMdzeNrt57Th5h2ofz5o0ynoxgHeVUARGS6iGSKSJaIzKljmmtEZKuIbBGRVz2GV4nIBvdjvsfwPiKy2r3M10UkrOmr4zsrs061/9sB4JYUHx3BsMQYxwpAWWUV9729iTlvf8mYvrG8N2siQ9pgE2CHiFBun9SPT3YcZs3uI07HMQ6ptwCISDDwBHAhMAi4TkQG1ZgmGbgPmKCqg4E7PUafVNXh7sdlHsP/BDyiqv2Bo8AtTVoTH1uZXUBcVDgpCVFORwk4U1Lj+WLvUY4cL2/R9z1QdJJrnlnFa5/v44eT+/HS90fTqb1f/S7xqRvH9yIuKpyHFmb6zTEX07K82QMYDWSpao6qlgNzgctrTPMD4AlVPQqgqmf8+Sauc+emAG+6B70MzGhA7malqqzILmR8v86t7jS/tmBqagKqsDSz5fYCVuUUculjy8k6VMxT3x3JvdNT2/ytqSPDQrhjSn8+33WEZTsLnI5jHCD1VX4RuRqYrqq3ul9fD4xR1Vke07wD7AAmAMHAb1T1Q/e4SmADUAn8UVXfEZE4YJX71z8ikgR8oKpDann/mcBMgISEhLS5c+c2akVLSkqIivLu13xeSTW/XH6S7w8JY1Ji894AriG5WpKTuapVuXvpSVI6BfGj4d8848bXuVSVRXsqmZtZTnw74Y6REfSIavihsdb6/1hRrcz59CQdwoT7x0W02A+e1vp5OaWpudLT09ep6qjTRqjqGR/A1cBzHq+vBx6vMc0CYB4QCvQB9gEd3eN6uP/tC+wG+gFxuPYqTs2fBGyuL0taWpo2VkZGhtfTvrg8R3vdu0D3Fh5v9Pt5qyG5WpLTuX7+74065P4Ptbyy6hvDfZnrRFml/uS1L7TXvQv01pfXaNHJ8kYvy+nPqy7e5Hr9873a694F+uHmA80fyK01f15OaGouYK3Wsk315qdOnnsDfUqie5inXGC+qlao6i5cewPJ7gKT5/43B1gKjAAKgY4iEnKGZTpmRXYhSbHtSIqNdDpKwJoyMJ7iskrW7j7aLMvfW3iCK55cwfyN+7nn/BSe+V5awN7u+8qRPegb156HP9pBVbUdCwgk3hSANUCy+6ydMOBaYH6Nad4BJgO4m3dSgBwR6SQi4R7DJwBb3RUpA9feBcCNwLtNWxXfqKyqZlVOod3902ET+8cRFhzEkma4KnhpZj6XPr6c/cdO8sJNZ3PH1OQzdqvY1oUEB3Hn+SlkHipmwab9TscxLajeAqCqlcAsYCGwDXhDVbeIyAMicuqsnoVAoYhsxbVhn62qhcBAYK2IbHQP/6OqbnXPcy9wt4hkAZ2B5325Yo21Zf9XFJdWMt7u/+Oo9uEhjO3XmcU+PB20ulp5fMlOvv/SGrrFRPDeHRNJHxDvs+W3ZpcM7UZq12geWbSDiqpqp+OYFuJVfwCq+j7wfo1h93s8V+Bu98NzmpXA0DqWmYPrDCO/siLbdTaE3f/HeVNT4/n1/C3sKjhOn7j2TVpWcWkF97yxkY+2HuKys7rzx6uGEhlm3WGcEhQk3DNtAD/4x1reWpfLtaN7Oh3JtAC7EriGlVmFpHaNtm7z/MCUVNev86ZeFJaVX8zlT6xg8fZ8fnXJIP567XDb+NfivIHxnJXUkb8u3unXt+Q2vmMFwENpRRVrdh9hnP369wtJsZEkx0eR0YQC8OHmA1z++AqKTlTwr1vGcMvEPnZtRx1EhNnTBnCgqJRXV+91Oo5pAVYAPHyx9yhlldV2ANiPTBkYz+pdhRSXVjRovqpq5cEPt3P7v76gf0I0C34y0Qq7Fyb078zYvrE8uTSLE+WVTscxzcwKgIfPsgsJDhLG9I11Oopxm5qaQEWVsrwBV6oePV7OTS9+zpNLs7n27CTeuG2s9YHrJRFh9gUDKCgp58UVu52OY5qZFQAPK7IKGJYYQ3SAng/uj0b27EhMu1CvzwbanFfEpY8vZ3XOEf5w5VD+eNUwwkNaR5eN/iKtVyxTUuN55pNsik42bM/LtC5WANyKSyvYmFtkzT9+JiQ4iMkDupCxPZ/qei5Smrc+l6ueWklllfL6bWO5zs5kabR7pqXwVWklzy3LcTqKaUZWANw+33WEqmplvN3/3+9MSY2n8Hg5G3OP1Tq+oqqa38zfwl2vb+SspI68d8dERvRsnV02+ovB3WO4eGg3Xli+i8KSMqfjmGZiBcBtRVYh4SFBjLQNh9+ZlNKFIKHWs4Hyi0v57t9X89LK3dw8oQ+v3DqGLtF2Cq8v3HV+CicrqnhqabbTUUwzsQLgtjK7gFG9OxERau3F/qZjZBijesWedhxg3Z6jXPrYcjblHeOv1w7n/ksHEWrdd/pM//gorhiRyD9W7eFgUanTcUwzsL8WXB1/bz9YbL1/+bEpA+PZsv8rjpZWo6q8snoP1z77GWEhQbz9wwlcPryH0xHbpDvPS0ZVeWzJTqejmGZgBQDX6Z8AE+z+P35rqvuq4LUHq7j3rU38ct5mxvWL471ZExnUvYPD6dqupNhIvn12Eq+v2cfewhNOxzE+ZgUAV/NPdEQIQ2xD4rf6x0eRFNuOV7eX88baXGal9+fFm86mY2Tb7bLRX9wxJZngIOHRj3c4HcX4mBUAXAeAx/TpTIi1H/stEeGys7oTHgzPXJ/Gzy4Y0Oa7bPQXCR0iuGFcL+ZtyGPnoWKn4xgfCvgt3r4jJ9h75AQT7PRPv3f3+QN4bGokFwzu6nSUgPPDyf2JDA3m4UW2F9CWBHwBsPb/1iM4SAi1X/2OiG0fxi3n9OWDzQfZnFfkdBzjIwFfAFZkF9AlOpzkeP/rCNoYf3LrOX2IaRfKnz/KdDqK8ZGALgCqysrsQsb362y3CDamHh0iQrl9Uj+WZh5m7e4jTscxPhDQBWBnfgmHi8us9y9jvHTj+F7ERYXz0MJMXB0BmtYsoAvAiqxT3T9a+78x3ogMC2FWej9W7zrC8izvb9Ft/FNAF4CV2YX0jI0kKTbS6SjGtBrXjelJj47t+LPtBbR6AVsAKquqWZVTaKd/GtNA4SHB/GRqfzbmFrFo6yGn45gm8KoAiMh0EckUkSwRmVPHNNeIyFYR2SIir7qHDReRz9zDNonItz2mf0lEdonIBvdjuE/WyEub939FcWmlNf8Y0whXjUykT1x7/vLRDqrq6afB+K96C4CIBANPABcCg4DrRGRQjWmSgfuACao6GLjTPeoEcIN72HTgURHp6DHrbFUd7n5saOK6NMip9n/rJ9aYhgsJDuLO85LJPFTMgk37nY5jGsmbPYDRQJaq5qhqOTAXuLzGND8AnlDVowCqmu/+d4eq7nQ/3w/kA118Fb4pVmYXkNo1mrgou3e8MY1x6bDupHaN5pFFO6ioqnY6jmkEqe8gjohcDUxX1Vvdr68HxqjqLI9p3gF2ABOAYOA3qvphjeWMBl4GBqtqtYi8BIwDyoDFwBxVPa3rIRGZCcwESEhISJs7d26jVrSkpISoKNfFXuVVyo8XnyA9KYTvDHS2AHjm8ieWq2ECNdf6/Er++kUZ3x8cxqQk7/vSDtTPq7Gamis9PX2dqo46bYSqnvEBXA085/H6euDxGtMsAOYBoUAfYB/Q0WN8NyATGFtjmADhuArD/fVlSUtL08bKyMj4+vmKrMPa694F+vHWg41enq945vInlqthAjVXdXW1Xvb4ch33fx9raUWl1/MF6ufVWE3NBazVWrap3jQB5QFJHq8T3cM85QLzVbVCVXfh2htIBhCRDsB/gF+q6iqPwnPAna0MeBFXU1OLWJlVSHCQMLpPbEu9pTFtkogwe9oA9heV8urqvU7HMQ3kTQFYAySLSB8RCQOuBebXmOYdYDKAiMQBKUCOe/p5wD9U9U3PGUSkm/tfAWYAmxu9Fg20IruAsxJjiI7wfpfVGFO7Cf07M7ZvLE9kZHGivNLpOKYB6i0AqloJzAIWAtuAN1R1i4g8ICKXuSdbCBSKyFYgA9fZPYXANcC5wE21nO75ioh8CXwJxAG/9+WK1aW4tIJNuUV2+qcxPiIizL5gAAUl5by0crfTcUwDhHgzkaq+D7xfY9j9Hs8VuNv98JzmX8C/6ljmlIaG9YXPdx2hqloZbxeAGeMzab1iSR/QhWc+yeG7Y3oR0872rluDgLsSeEVWIeEhQYzs2cnpKMa0KfdMG0DRyQqeX5bjdBTjpYArACuzCzi7dywRocFORzGmTRnSI4aLhnbl+eW7KCw57Yxu44cCqgAUlJSx/WCxNf8Y00zuPj+FkxVVPLU02+koxgsBVQBWurt/tAPAxjSP/vHRzBjRg3+s2sPBolKn45h6BFQB+Cy7gOiIEIb2iHE6ijFt1l3npaCqPLZkp9NRTD0CqgCsyCpkbN/OBFvH4sY0m6TYSL59dhKvr9nH3sITTscxZxAwBeDwiWr2HjnBBLv7pzHN7o4pyQQHCY8u3uF0FHMGAVMAth6pAmBCf2v/N6a5JXSI4IZxvXhnfR5Z+cVOxzF1CJgCsK2wii7R4fSP9787/RnTFv1wcn/ahQbz8CLbC/BXAVEAVJWthdWM79cZ162HjDHNLbZ9GLdM7MP7Xx5kc16R03FMLQKiAOzML+GrcmWCnf5pTIu69dy+xLQL5S8fZTodxdQiIArAqe4f7QIwY1pWh4hQbpvUl4zMw6zdfcTpOKaGACkAhcRHComdIp2OYkzAuWl8b+KiwnloYeapzqCMnwiIAnDDuF5cnRzmdAxjAlJkWAg/Tu/H6l1HWO7eGzf+ISAKwLkpXRjdzas7XxtjmsF3xvSke0wEf7a9AL8SEAXAGOOs8JBgfnpeMhtzi1i09ZDTcYybFQBjTIu4amQifeLa8/CiHVTbXoBfsAJgjGkRIcFB3HleMtsPFvP5gSqn4xisABhjWtClw7qT2jWaeVnlVFZVOx0n4FkBMMa0mKAg4e7zUzh0Qnnri1yn4wQ8rwqAiEwXkUwRyRKROXVMc42IbBWRLSLyqsfwG0Vkp/txo8fwNBH50r3Mv4ndo8GYgHD+oAT6xgTxt8VZlFVaU5CT6i0AIhIMPAFcCAwCrhORQTWmSQbuAyao6mDgTvfwWODXwBhgNPBrETnVG/tTwA+AZPdjug/Wxxjj50SEK5PDyDt2ktdW73U6TkDzZg9gNJClqjmqWg7MBS6vMc0PgCdU9SiAqua7h18ALFLVI+5xi4DpItIN6KCqq9R1UvA/gBlNXx1jTGswuHMQY/rE8nhGNifKK52OE7C8KQA9gH0er3PdwzylACkiskJEVonI9Hrm7eF+fqZlGmPaKBFh9gUDKCgp46WVu52OE7B8dXlsCK5mnMlAIvCpiAz1xYJFZCYwEyAhIYGlS5c2ajklJSWNnrc5Wa6GsVwN48+52L2JYXHBPLE4k14V+2gf6vxhQH/+vJojlzcFIA9I8nid6B7mKRdYraoVwC4R2YGrIOThKgqe8y51D0+sZ5kAqOqzwLMAo0aN0smTJ9c2Wb2WLl1KY+dtTparYSxXw/h7rrjkIi55bDmZ2p27Jw9wOpbff16+5k0T0BogWUT6iEgYcC0wv8Y07+De0ItIHK4moRxgITBNRDq5D/5OAxaq6gHgKxEZ6z775wbgXR+sjzGmFRnSI4aLhnbl+eW7KCwpczpOwKm3AKhqJTAL18Z8G/CGqm4RkQdE5DL3ZAuBQhHZCmQAs1W1UFWPAL/DVUTWAA+4hwH8CHgOyAKygQ98uF7GmFbi7vNTOFlRxdOfZDsdJeB4dQxAVd8H3q8x7H6P5wrc7X7UnPcF4IVahq8FhjQwrzGmjekfH82MET34x2d7uPWcviR0iHA6UsCwK4GNMY67c2oKVdXKY0t2Oh0loFgBMMY4rmfnSL59dhJzP9/HviMnnI4TMKwAGGP8wh1TkgkOEh792PYCWooVAGOMX+gaE8H1Y3sxb30uWfnFTscJCFYAjDF+44eT+9EuNJiHF+1wOkpAsAJgjPEbnaPCuXliH97/8iCb84qcjtPmWQEwxviVW8/pS0y7UP7yUabTUdo8KwDGGL8S0y6U2yb1JSPzMOv2HKl/BtNoVgCMMX7npvG9iYsK56GFmah1IN9srAAYY/xOZFgIP07vx6qcI6zIKnQ6TptlBcAY45e+M6Yn3WMieOgj2wtoLlYAjDF+KTwkmJ9MTWbjvmN8vC2//hlMg1kBMMb4ravSEundOZK/fJRJdbXtBfiaFQBjjN8KDQ7irvNT2H6wmAVfHnA6TptjBcAY49cuHdadAQnRPLJoB5VV1U7HaVOsABhj/FpQkHD3tBR2FRznrS9ynY7TplgBMMb4vWmDEjgrMYa/Lc6irLLK6ThthhUAY4zfExF+dsEA8o6d5LXVe52O02ZYATDGtAoT+8cxpk8sj2dkc6K80uk4bYIVAGNMqyAizL5gAAUlZby8co/TcdoEKwDGmFZjVO9YJg/owtOfZPNVaYXTcVo9rwqAiEwXkUwRyRKRObWMv0lEDovIBvfjVvfwdI9hG0SkVERmuMe9JCK7PMYN9+WKGWPapp9NG0DRyQqeW7bL6SitXkh9E4hIMPAEcD6QC6wRkfmqurXGpK+r6izPAaqaAQx3LycWyAI+8phktqq+2fj4xphAM6RHDBcO6crzy3K4aXxvYtuHOR2p1fJmD2A0kKWqOapaDswFLm/Ee10NfKCqJxoxrzHGfO3u81M4UVHF059kOx2lVfOmAPQA9nm8znUPq+kqEdkkIm+KSFIt468FXqsx7H/d8zwiIuHeRTbGBLrkhGiuGN6Dl1fu5tBXpU7HabWkvtusisjVwHRVPdWufz0wxrO5R0Q6AyWqWiYitwHfVtUpHuO7AZuA7qpa4THsIBAGPAtkq+oDtbz/TGAmQEJCQtrcuXMbtaIlJSVERUU1at7mZLkaxnI1TFvOlX+imvuWnWRSUgg3DPLN78e2+nmlp6evU9VRp41Q1TM+gHHAQo/X9wH3nWH6YKCoxrCfAs+eYZ7JwIL6sqSlpWljZWRkNHre5mS5GsZyNUxbz/WLtzdpnzkL9M6563Xr/qImL6+tfl7AWq1lm1rvQWBgDZAsIn2APFxNOd/xnEBEuqnqqVv1XQZsq7GM69yF47R5RESAGcBmL7IYY8zX5lyYSnhIMHPX7GXe+jzOTenC7ef2ZVy/zrg2LeZM6i0AqlopIrOAhbh+3b+gqltE5AFcVWU+8BMRuQyoBI4AN52aX0R6A0nAJzUW/YqIdAEE2ADc3uS1McYElOiIUO6/dBA/nZrMv1bv4cUVu/nOc6sZ2iOG2yb1ZfrgroQE2+VOdfFmDwBVfR94v8aw+z2e30eNX/ge43ZTy0Fj9ThGYIwxTRETGcqP0/tzy8Q+zFufx98/zWHWq+vpGRvJD87pw9VpSbQLC3Y6pt+x0miMaTMiQoO5bnRPFt09iae/l0bnqDB+9e4WJvxpCY9+vIMjx8udjuhXvNoDMMaY1iQ4SJg+pCsXDE5g7Z6jPPNJNo9+vJOnP8nm26OSuPWcviTFRjod03FWAIwxbZaIcHbvWM7uHcvOQ8U8+2kOr36+l3+u2sNFQ7tx27n9GJoY43RMx1gBMMYEhOSEaB761lncM20AL67cxaur9rJg0wEm9O/MzHP7cW5ynNMRW5wVAGNMQOkaE8F9Fw7kx+n9eW31Xl5YsYsbX/icgd06cE6XSiZUVRMaIGcOBcZaGmNMDR0iQrltUj8+/Xk6D149jMqqap7dVMbkh5bywvJdHC9r+53OWAEwxgS08JBgrhmVxMI7z+XOkeH06NiOBxZsZfwfl/DnhZkcLi5zOmKzsQJgjDFAUJAwPD6EN24fx1s/HM/YvrE8sTSLCX9awi/mfcmuguNOR/Q5OwZgjDE1pPXqxDPXjyLncAl/X7aLN9fl8trne7lgUFdum9SXET07OR3RJ6wAGGNMHfp2ieIPVw7lrvOTeXnlbv752R4+3HKQ0X1iuX1SXyanxBMU1HrvOWRNQMYYU4/46AhmX5DKyvum8qtLBpF75AQ3v7SW6X/9lDfX5VJeWe10xEaxAmCMMV6KCg/hlol9+OTn6Tzy7bMIEuFn/97IuQ9m8Oyn2RS3so7qrQAYY0wDhQYHccWIRD746Tm8fPNo+nZpz/+9v53xf1jCHz/Y3mp6KbNjAMYY00giwqSULkxK6cKm3GM882kOz36azfPLc7hiRA9mntuX/vHRTseskxUAY4zxgWGJHXniOyPZU3ic55bt4t/r9vHG2lzOG5jA7ZP6Mqp3rNMRT2NNQMYY40O9OrfndzOGsOLeKfx0ajLr9hzh6qc/48onV7Bwy0Gqq8/cD3tLsgJgjDHNoHNUOHedn8KKOVP47WWDyS8u47Z/ruO8Rz5h7ud7Ka2ocjqiFQBjjGlOkWEh3Di+N0t/NpnHrhtBZFgwc97+knMezOCJjCyKTjh35pAdAzDGmBYQEhzEpWd155Jh3ViZXcjTn2Tz0MJMnszI4rrRPbl5Yh+6d2zXspla9N2MMSbAiQgT+scxoX8cW/YX8fdPc3hx5W5eWrmby87qzsxJfUnt2qFFslgTkDHGOGRw9xgevXYEn8yezPXjevHB5oNMf3QZN734OZ9lF6LavAeMvSoAIjJdRDJFJEtE5tQy/iYROSwiG9yPWz3GVXkMn+8xvI+IrHYv83URCfPNKhljTOuS2CmSX186mJVzpnDP+Sl8mVvEdX9fxYwnVvD+lweobqZCUG8BEJFg4AngQmAQcJ2IDKpl0tdVdbj78ZzH8JMewy/zGP4n4BFV7Q8cBW5p/GoYY0zr16l9GHdMTWbFnCn87xVDKDpZwY9e+YI5y06y/eBXPn8/b/YARgNZqpqjquXAXODyprypiAgwBXjTPehlYEZTlmmMMW1FRGgw3x3Ti8X3TOap744kPjKInrGRPn8fbwpAD2Cfx+tc97CarhKRTSLypogkeQyPEJG1IrJKRGa4h3UGjqnqqT7X6lqmMcYErOAg4cKh3fjZqAgiw3x/zo7Ud5BBRK4Gpqvqre7X1wNjVHWWxzSdgRJVLROR24Bvq+oU97geqponIn2BJcBUoAhY5W7+wV0wPlDVIbW8/0xgJkBCQkLa3LlzG7WiJSUlREVFNWre5mS5GsZyNYzlapi2mis9PX2dqo46bYSqnvEBjAMWery+D7jvDNMHA0V1jHsJuBoQoAAIqe096nqkpaVpY2VkZDR63uZkuRrGcjWM5WqYtpoLWKu1bFO9aQJaAyS7z9oJA64F5ntOICLdPF5eBmxzD+8kIuHu53HABGCrO1CGuxgA3Ai860UWY4wxPlJvo5KqVorILGAhrl/3L6jqFhF5AFdVmQ/8REQuAyqBI8BN7tkHAs+ISDWu4w1/VNWt7nH3AnNF5PfAeuB5H66XMcaYenh1VEFV3wferzHsfo/n9+FqGqo530pgaB3LzMF1hpExxhgH2JXAxhgToKwAGGNMgLICYIwxAare6wD8iYgcBvY0cvY4XKee+hvL1TCWq2EsV8O01Vy9VLVLzYGtqgA0hYis1douhHCY5WoYy9UwlqthAi2XNQEZY0yAsgJgjDEBKpAKwLNOB6iD5WoYy9UwlqthAipXwBwDMMYY802BtAdgjDHGgxUAY4wJUAFRAEQkWETWi8gCp7N4EpGO7g50tovINhEZ53QmABG5S0S2iMhmEXlNRCIcyvGCiOSLyGaPYbEiskhEdrr/7eQnuR5y/z9uEpF5ItLRH3J5jLtHRNR9V16/yCUid7g/sy0i8qA/5BKR4e7Oqza4O7Jq8fuViUiSiGSIyFb3Z/NT93Cff/cDogAAP8V9i2o/81fgQ1VNBc7CDzKKSA/gJ8AodXXQE4zrFuBOeAmYXmPYHGCxqiYDi92vW9pLnJ5rETBEVYcBO6jl5ogt4CVOz3Wqw6VpwN6WDuT2EjVyiUg6rq5lz1LVwcCf/SEX8CDwW1UdDtzvft3SKoF7VHUQMBb4sbsfdp9/99t8ARCRROBi4Ln6pm1JIhIDnIv7NtiqWq6qxxwN9V8hQDsRCQEigf1OhFDVT3HdXtzT5bj6kAaH+pKuLZeqfqT/7eJ0FZDoD7ncHgF+DjhyxkcduX6I6/bwZe5p8v0klwId3M9jcOC7r6oHVPUL9/NiXD8Me9AM3/02XwCAR3F9+asdzlFTH+Aw8KK7eeo5EWnvdChVzcP1a2wvcABX724fOZvqGxJU9YD7+UEgwckwdbgZ+MDpEAAicjmQp6obnc5SQwpwjoisFpFPRORspwO53Qk8JCL7cP0dOLEn9zUR6Q2MAFbTDN/9Nl0AROQSIF9V1zmdpRYhwEjgKVUdARzHmeaMb3C3K16Oq0B1B9qLyPecTVU7d89yfnUes4j8Etcu/Ct+kCUS+AWupgx/EwLE4mrimA28ISLibCTAtWdyl6omAXfhYEdVIhIFvAXcqapfeY7z1Xe/TRcAXF1QXiYiu4G5wBQR+Zezkb6WC+Sq6mr36zdxFQSnnQfsUtXDqloBvA2MdziTp0OnuiB1/9viTQd1EZGbgEuA76p/XGDTD1ch3+j+G0gEvhCRro6mcskF3nZ3Wfs5rj30Fj9AXYsbcX3nAf6NQ51WiUgoro3/K6p6Ko/Pv/ttugCo6n2qmqiqvXEdyFyiqn7xa1ZVDwL7RGSAe9BUYOsZZmkpe4GxIhLp/kU2FT84OO1hPq4/UvCjvqRFZDqupsbLVPWE03kAVPVLVY1X1d7uv4FcYKT7u+e0d4B0ABFJAcLwj7tw7gcmuZ9PAXa2dAD3393zwDZVfdhjlO+/+7X1FN8WH8BkYIHTOWpkGg6sBTbh+oPo5HQmd67fAtuBzcA/gXCHcryG6zhEBa6N1y1AZ1xnQOwEPgZi/SRXFrAP2OB+PO0PuWqM3w3E+UMuXBv8f7m/Y18AU/wk10RgHbARV7t7mgO5JuJq3tnk8X26qDm++3YrCGOMCVBtugnIGGNM3awAGGNMgLICYIwxAcoKgDHGBCgrAMYYE6CsABhjTICyAmCMMQHq/wFLQ2zoOUtFpQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "f,ax = plt.subplots()\n",
    "\n",
    "pd.Series(v_coh, index=[len(topic) for topic in l_topics]).plot(ax=ax)\n",
    "ax.grid()\n",
    "#ax.set_ylim(0.2,0.7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LDA sur l'ensemble des donnée"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<module 'grand_debat.theme_detection' from '/Users/charlesprat/RepoGit/grand-debat/grand_debat/theme_detection.py'>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "importlib.reload(td)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "# Plusieurs essais random me montrent que sur la question sélectionnée UXVlc3Rpb246MTQ3\n",
    "# le premier maximum de la coherence se trouve très souvent entre 5 et 6. \n",
    "\n",
    "n_topics = 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "topic_detector = td.GDebatTopicDetection(gd_prep, n_topics, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iteration: 1 of max_iter: 100\n",
      "iteration: 2 of max_iter: 100\n",
      "iteration: 3 of max_iter: 100\n",
      "iteration: 4 of max_iter: 100, perplexity: 786.4160\n",
      "iteration: 5 of max_iter: 100\n",
      "iteration: 6 of max_iter: 100\n",
      "iteration: 7 of max_iter: 100\n",
      "iteration: 8 of max_iter: 100, perplexity: 699.7166\n",
      "iteration: 9 of max_iter: 100\n",
      "iteration: 10 of max_iter: 100\n",
      "iteration: 11 of max_iter: 100\n",
      "iteration: 12 of max_iter: 100, perplexity: 680.1894\n",
      "iteration: 13 of max_iter: 100\n",
      "iteration: 14 of max_iter: 100\n",
      "iteration: 15 of max_iter: 100\n",
      "iteration: 16 of max_iter: 100, perplexity: 671.9533\n",
      "iteration: 17 of max_iter: 100\n",
      "iteration: 18 of max_iter: 100\n",
      "iteration: 19 of max_iter: 100\n",
      "iteration: 20 of max_iter: 100, perplexity: 667.9805\n",
      "iteration: 21 of max_iter: 100\n",
      "iteration: 22 of max_iter: 100\n",
      "iteration: 23 of max_iter: 100\n",
      "iteration: 24 of max_iter: 100, perplexity: 666.3586\n",
      "iteration: 25 of max_iter: 100\n",
      "iteration: 26 of max_iter: 100\n",
      "iteration: 27 of max_iter: 100\n",
      "iteration: 28 of max_iter: 100, perplexity: 665.2037\n",
      "iteration: 29 of max_iter: 100\n",
      "iteration: 30 of max_iter: 100\n",
      "iteration: 31 of max_iter: 100\n",
      "iteration: 32 of max_iter: 100, perplexity: 664.6420\n",
      "iteration: 33 of max_iter: 100\n",
      "iteration: 34 of max_iter: 100\n",
      "iteration: 35 of max_iter: 100\n",
      "iteration: 36 of max_iter: 100, perplexity: 664.3124\n",
      "LDA visualization in pyLDAVIS_tf.html\n"
     ]
    }
   ],
   "source": [
    "topic_detector.compute_topic_detection(data_bow=gd_prep.answ_bow, LDAVis=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['hiver', 'été', 'chaud', 'année', 'neige']\n",
      "['saison', 'disparition', 'insecte', 'oiseau', 'espèce']\n",
      "['consommation', 'voiture', 'transport', 'produit', 'énergie']\n",
      "['sécheresse', 'inondation', 'canicule', 'tempête', 'fréquent']\n",
      "['pollution', 'air', 'santé', 'allergie', 'problème']\n",
      "['chaleur', 'pollution', 'pic', 'ville', 'fort']\n",
      "['enfant', 'monde', 'avenir', 'planète', 'réchauffement']\n",
      "['température', 'eau', 'augmentation', 'sécheresse', 'hausse']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "for topic in topic_detector.get_topics_by_relevance(lambd=1):\n",
    "    print(topic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interprétation et résumé"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<module 'grand_debat.text_summarization' from '/Users/charlesprat/RepoGit/grand-debat/grand_debat/text_summarization.py'>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import grand_debat.text_summarization as ts\n",
    "importlib.reload(ts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence segmentation\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 92795/92795 [02:04<00:00, 742.63it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Topic detection\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 169480/169480 [01:54<00:00, 1486.29it/s]\n"
     ]
    }
   ],
   "source": [
    "gd_summ = ts.GDebatSummerization(answers, topic_detector)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "titles = gd_summ.create_titles()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[\"On voit les températures grimpées d'années en années : de moins en moins de neige l'hiver, de plus en plus de canicules l'été, la sécheresse au moins la moitié de l'année...\",\n",
       " \"Raréfaction voire disparition d'espèces animales (insectes et par conséquences oiseaux, et autres animaux) autochtones et nivellement des saisons\",\n",
       " \"Les produits offerts aujourd'hui sur le marché nécessitent une  consommation  toujours plus   grande d'énergie donc d'électricité donc le recours au nucléaire\",\n",
       " 'Les épisodes de plus en plus fréquents  de catastrophes naturelles (sécheresses, canicules, inondations, tempêtes.....)',\n",
       " \"réchauffement climatique est lié à la pollution de l'air qui entraine des problèmes de santé tel que allergie aux pollens et asthme allergique cancers\",\n",
       " 'Habitant d’une grande ville, je constate que les épisodes de forte chaleur et de canicule sont de plus en plus fréquents et de plus en plus longs.',\n",
       " \"Réflexion sur la responsabilité engagée envers les générations à venir..bien égoîstement en priorité sur l'avenir de mes enfants et petits enfants dans ce monde.\",\n",
       " \"bouleversement des écosystèmes engendré par la hausse des températures, impacte sur les réserves d'eau, augmentation sensible du niveau de la mer\"]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "index_by_topics = gd_summ.group_sentences_by_topics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[\"Nourriture non saine, eau intoxiquée (les micro organismes nocifs tels que le glyphosate, les métaux lourds, résidus de plastiques sont présents même dans l'eau du robinet, il ne faut pas se leurrer), terre polluée, air pollué...\",\n",
       " \"Nourriture non saine, eau intoxiquée (les micro organismes nocifs tels que le glyphosate, les métaux lourds, résidus de plastiques sont présents même dans l'eau du robinet), terre polluée, air pollué...\",\n",
       " \"la pollution de l'air affecte ma respiration, la présence de produits chimiques (pesticides, perturbateurs endocriniens) dans mon alimentation a impacté ma santé (cancer).\",\n",
       " 'La pollution impacte ma santé : irritation des yeux, maux de tête, maux de gorge les jours de grande pollution ….',\n",
       " \"La qualité de l'air est très mauvaise à Paris et en proche banlieue ce qui impact fortement la santé (maux de tête, poumons, irritations cutanées, etc.).\",\n",
       " \"Dés que l on est dans une grande ville, on sent directement l'air pollué ce qui peut entrainer des problèmes de santé (pbs respiratoires , maux de tête...\",\n",
       " \"Le problème n'étant pas un changement mais la pollution elle-même: oui je respire un air impropre tous les jours, je me baigne dans des eaux polluées et je mange des produits venant d'une terre ou d'une eau contaminée!\",\n",
       " \"réchauffement climatique est lié à la pollution de l'air qui entraine des problèmes de santé tel que allergie aux pollens et asthme allergique cancers\",\n",
       " \"Problèmes de santé (peau, digestif, respiratoire) à cause de la pollution de l'air et la toxicité dans les produits.\",\n",
       " \"De plus en plus de personnes sont touchées par les maladies pulmonaires (asthme, allergies respiratoires) dues aux microparticules et autres polluants de l'air, principalement en ville.\"]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "np.array(gd_summ.answ_sents)[index_by_topics[4][:10]].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "9588"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(index_by_topics[4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "959"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lidx = index_by_topics[4][::10]\n",
    "len(lidx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "import networkx as nx\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "nb_sentences = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/charlesprat/miniconda3/envs/gd/lib/python3.8/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
      "  and should_run_async(code)\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-36-835ff2f625b9>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mlogging\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetLogger\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetLevel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlogging\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mINFO\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0md2v_model\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mDoc2Vec\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtagged_data\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvector_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m50\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m40\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mworkers\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/miniconda3/envs/gd/lib/python3.8/site-packages/gensim/models/doc2vec.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, documents, corpus_file, vector_size, dm_mean, dm, dbow_words, dm_concat, dm_tag_count, dv, dv_mapfile, comment, trim_rule, callbacks, window, epochs, shrink_windows, **kwargs)\u001b[0m\n\u001b[1;32m    292\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvectors_lockf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mones\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mREAL\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# 0.0 values suppress word-backprop-updates; 1.0 allows\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    293\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 294\u001b[0;31m         super(Doc2Vec, self).__init__(\n\u001b[0m\u001b[1;32m    295\u001b[0m             \u001b[0msentences\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcorpus_iterable\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    296\u001b[0m             \u001b[0mcorpus_file\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcorpus_file\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/gd/lib/python3.8/site-packages/gensim/models/word2vec.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, sentences, corpus_file, vector_size, alpha, window, min_count, max_vocab_size, sample, seed, workers, min_alpha, sg, hs, negative, ns_exponent, cbow_mean, hashfxn, epochs, null_word, trim_rule, sorted_vocab, batch_words, compute_loss, callbacks, comment, max_final_vocab, shrink_windows)\u001b[0m\n\u001b[1;32m    424\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_check_corpus_sanity\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcorpus_iterable\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcorpus_iterable\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcorpus_file\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcorpus_file\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpasses\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mepochs\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    425\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuild_vocab\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcorpus_iterable\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcorpus_iterable\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcorpus_file\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcorpus_file\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrim_rule\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtrim_rule\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 426\u001b[0;31m             self.train(\n\u001b[0m\u001b[1;32m    427\u001b[0m                 \u001b[0mcorpus_iterable\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcorpus_iterable\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcorpus_file\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcorpus_file\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtotal_examples\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcorpus_count\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    428\u001b[0m                 \u001b[0mtotal_words\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcorpus_total_words\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mepochs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstart_alpha\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0malpha\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/gd/lib/python3.8/site-packages/gensim/models/doc2vec.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, corpus_iterable, corpus_file, total_examples, total_words, epochs, start_alpha, end_alpha, word_count, queue_factor, report_delay, callbacks, **kwargs)\u001b[0m\n\u001b[1;32m    512\u001b[0m             \u001b[0mkwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'start_doctags'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstart_doctags\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    513\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 514\u001b[0;31m         super(Doc2Vec, self).train(\n\u001b[0m\u001b[1;32m    515\u001b[0m             \u001b[0mcorpus_iterable\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcorpus_iterable\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcorpus_file\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcorpus_file\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    516\u001b[0m             \u001b[0mtotal_examples\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtotal_examples\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtotal_words\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtotal_words\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/gd/lib/python3.8/site-packages/gensim/models/word2vec.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, corpus_iterable, corpus_file, total_examples, total_words, epochs, start_alpha, end_alpha, word_count, queue_factor, report_delay, compute_loss, callbacks, **kwargs)\u001b[0m\n\u001b[1;32m   1067\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1068\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mcorpus_iterable\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1069\u001b[0;31m                 trained_word_count_epoch, raw_word_count_epoch, job_tally_epoch = self._train_epoch(\n\u001b[0m\u001b[1;32m   1070\u001b[0m                     \u001b[0mcorpus_iterable\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcur_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcur_epoch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtotal_examples\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtotal_examples\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1071\u001b[0m                     \u001b[0mtotal_words\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtotal_words\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mqueue_factor\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mqueue_factor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreport_delay\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreport_delay\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/gd/lib/python3.8/site-packages/gensim/models/word2vec.py\u001b[0m in \u001b[0;36m_train_epoch\u001b[0;34m(self, data_iterable, cur_epoch, total_examples, total_words, queue_factor, report_delay, callbacks)\u001b[0m\n\u001b[1;32m   1428\u001b[0m             \u001b[0mthread\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1429\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1430\u001b[0;31m         trained_word_count, raw_word_count, job_tally = self._log_epoch_progress(\n\u001b[0m\u001b[1;32m   1431\u001b[0m             \u001b[0mprogress_queue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjob_queue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcur_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcur_epoch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtotal_examples\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtotal_examples\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1432\u001b[0m             \u001b[0mtotal_words\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtotal_words\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreport_delay\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreport_delay\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_corpus_file_mode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/gd/lib/python3.8/site-packages/gensim/models/word2vec.py\u001b[0m in \u001b[0;36m_log_epoch_progress\u001b[0;34m(self, progress_queue, job_queue, cur_epoch, total_examples, total_words, report_delay, is_corpus_file_mode)\u001b[0m\n\u001b[1;32m   1283\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1284\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0munfinished_worker_count\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1285\u001b[0;31m             \u001b[0mreport\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mprogress_queue\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# blocks if workers too slow\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1286\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mreport\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# a thread reporting that it finished\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1287\u001b[0m                 \u001b[0munfinished_worker_count\u001b[0m \u001b[0;34m-=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/gd/lib/python3.8/queue.py\u001b[0m in \u001b[0;36mget\u001b[0;34m(self, block, timeout)\u001b[0m\n\u001b[1;32m    168\u001b[0m             \u001b[0;32melif\u001b[0m \u001b[0mtimeout\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    169\u001b[0m                 \u001b[0;32mwhile\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_qsize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 170\u001b[0;31m                     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnot_empty\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    171\u001b[0m             \u001b[0;32melif\u001b[0m \u001b[0mtimeout\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    172\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"'timeout' must be a non-negative number\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/gd/lib/python3.8/threading.py\u001b[0m in \u001b[0;36mwait\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m    300\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m    \u001b[0;31m# restore state no matter what (e.g., KeyboardInterrupt)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    301\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mtimeout\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 302\u001b[0;31m                 \u001b[0mwaiter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0macquire\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    303\u001b[0m                 \u001b[0mgotit\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    304\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "from gensim.models.doc2vec import Doc2Vec, TaggedDocument\n",
    "tagged_data = [TaggedDocument(d, [i]) for i, d in enumerate(gd_summ.sents_lems)]\n",
    "\n",
    "logging.getLogger().setLevel(logging.INFO)\n",
    "d2v_model = Doc2Vec(tagged_data, vector_size=50, epochs=40, workers=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_lems = np.array(gd_summ.sents_lems)[lidx]\n",
    "\n",
    "sents_emb = [d2v_model.infer_vector(lems) for lems in tqdm(topic_lems)]\n",
    "sim_mat = cosine_similarity(sents_emb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f, ax = plt.subplots()\n",
    "\n",
    "img = ax.imshow(sim_mat)\n",
    "plt.colorbar(img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_corr = pd.Series(sim_mat.mean(axis=1)).sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_corr.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_topic_proba([\"érosion littoral\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_lems[max_corr.index[:10]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.array(gd_summ.answ_sents)[np.array(lidx)[max_corr.index[:10]]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx_graph = nx.from_numpy_array(sim_mat)\n",
    "scores = nx.pagerank(nx_graph)\n",
    "sub_idx = pd.Series(scores).sort_values(ascending=False).index[:nb_sentences]\n",
    "\n",
    "        #return np.array(gd_summ.answ_sents)[np.array(lidx)[sub_idx]].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d2v_model.dv[[0]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ilem = 100\n",
    "cs = [\n",
    "    cosine_similarity(d2v_model.dv[[ilem]],\n",
    "                  [d2v_model.infer_vector(gd_summ.sents_lems[ilem])])[0][0]\n",
    "    for ilem in tqdm(range(1000))\n",
    "]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f, ax = plt.subplots()\n",
    "ax.plot(cs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.Series(scores).plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d2v_title = gd_summ.d2v_model.infer_vector(gd_prep.tokenize([titles[4]])[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f, ax = plt.subplots()\n",
    "\n",
    "ax.plot(gd_summ.d2v_model.dv.cosine_similarities(\n",
    "    d2v_title, sents_emb))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_topic_proba(np.array(gd_summ.answ_sents)[np.array(lidx)[sub_idx]].tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.array(lidx)[sub_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_proba(texts):\n",
    "    return gd_summ.topic_detector.tf_lda.transform(\n",
    "        gd_prep.prepare_data(texts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summ = ['Impact sur les maisons (fissures dues aux mouvements de terrain)',\n",
    " \"Ce que j'ai vu personnellement est la fonte du glacier du Mont-Blanc et le recul du trait de côte.\",\n",
    " 'Les stations de moyenne altitude près de chez moi sont confrontées à une baisse de fréquentation.',\n",
    " ',  la prolifération de moustiques tigres, de frelons asiatiques',\n",
    " 'Pour constater aussi les épandages de désherbants, de pesticides par nos voisins agriculteurs .']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_topic_proba(summ)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_topic_proba(np.array(gd_summ.answ_sents)[index_by_topics[4][:5]].tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gd_prep.tokenize([titles[4]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_lems = np.array(gd_summ.sents_lems)[lidx]\n",
    "sents_emb = [gd_summ.d2v_model.infer_vector(lems) for lems in topic_lems]      "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "f, ax = plt.subplots()\n",
    "ax.plot(gd_summ.topic_proba[index_by_topics[4]][:,4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gd_summ.get_topic_summary(index_by_topics[4], 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "J'interprête \"à la main\" chacun des thèmes comme cela :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_names = [\"Biodiversité\", \"Pollution de l'air / santé\", \"Événements extrêmes\", \n",
    "               \"Réchauffement / saisons déréglées\", \n",
    "               \"Inquiétude avenir / impact sur nos dépenses\" ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_detector.tf_lda.transform(gd_prep.answ_bow)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "topics_pct = pd.Series(index=topic_names, \n",
    "                       data=(topic_detector.tf_lda.transform(gd_prep.answ_bow) > 1.0/n_topics).mean(axis=0)*100)\n",
    "\n",
    "f,ax = plt.subplots()\n",
    "topics_pct.sort_values().plot(kind=\"barh\")\n",
    "ax.grid()\n",
    "ax.set_xlabel(\"%\")\n",
    "ax.set_title(\"Pourcentage des réponses\\nmentionnant les thèmes suivants\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# topics OpinionWay \n",
    "# https://granddebat.fr/media/default/0001/01/b88758e8caa2733bec607a74b3b5371cc0a3b420.pdf \n",
    "topics_ow = pd.Series(\n",
    "    {\"Les épisodes de chaleur, les sécheresses\": 24.6,\n",
    "    \"Des événements climatiques plus fréquents\": 16.5,\n",
    "    \"L'impact lié à la qualité de l'air\": 12.4,\n",
    "    \"Autres impacts liés à la pollution\": 12.2,\n",
    "    \"L'impact lié à l'eau : pollution, inondations, consommation\": 9.3,\n",
    "    \"Les conséquences sur la santé\": 11.9,\n",
    "     \"L'impact lié à la faune, la flore, la biodiversité\": 8.4,\n",
    "     \"Autres éléments\": 3.2,\n",
    "     \"Autres mentions concernant l'environnement\": 2.4,\n",
    "     \"L'impact lié au littoral et à la mer\": 1.4,\n",
    "     \"L'impact lié aux dépenses, prix, taxes\": 1.3,\n",
    "     \"Trop de déchets, le problème de la gestion des déchets\": 1.3,\n",
    "     \"L'impact lié à l'agriculture\": 1.2,\n",
    "     \"L'impact lié aux transports\": 1.1,\n",
    "     \"L'immigration de réfugiés climatiques\": 0.6\n",
    "    }\n",
    ")\n",
    "\n",
    "f,ax = plt.subplots()\n",
    "topics_ow.sort_values().plot(kind=\"barh\",ax=ax)\n",
    "ax.grid()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "gd",
   "language": "python",
   "name": "gd"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}