03_topic_modeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook 3: Topic Modeling and Extraction\n",
    "\n",
    "##### Please refer to the Python Requirements and Installation Guide pdf \n",
    "\n",
    "####  Purpose: The Purpose of the code below is to dig deeper into the \"'Gut am Arbeitgeber finde ich_plain_text' and 'Schlecht am Arbeitgeber finde ich_plain_text' colums and analyse the reviews. For this purpose, we utilise the Latent Dirichlet Allocation(LDA Model) for topic modelling and key topic extraction. LDA topic modeling is a probabilistic modeling method for classifying documents based on information related to the topic of the text in a large number of unstructured documents. By analysing the 2 columns/features, we are able to better understand the most spoken about topics within these columns.\n",
    "\n",
    "#### Intuition: Topic modeling is a type of statistical modeling for discovering the abstract “topics” that occur in a collection of documents. Latent Dirichlet Allocation (LDA) is an example of topic model and is used to classify text in a document to a particular topic. It builds a topic per document model and words per topic model, modeled as Dirichlet distributions. Latent Dirichlet Allocation (LDA) is an unsupervised clustering technique in which words are represented as topics, and documents are represented as a collection of these word topics.\n",
    "\n",
    "#### Additional Python Libraries Required: \n",
    "\n",
    "1. __Spacy__ <br> \n",
    "Link: https://spacy.io/ <br>\n",
    "!pip install -U spacy <br>\n",
    "!python -m spacy download en_core_web_sm\n",
    "\n",
    "2. __Gensim__ <br> \n",
    "Link: https://pypi.org/project/gensim/ <br>\n",
    "!pip install gensim <br> \n",
    "\n",
    "3. __pyLDAvis__ <br> \n",
    "Link: https://pypi.org/project/pyLDAvis/ <br>\n",
    "!pip install pyLDAvis <br> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing packages that will be required: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# imports: \n",
    "import pandas as pd\n",
    "from sklearn.feature_extraction import text\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "import gensim\n",
    "import numpy as np\n",
    "from gensim.utils import simple_preprocess\n",
    "import gensim.corpora as corpora\n",
    "import spacy\n",
    "import pyLDAvis\n",
    "import pyLDAvis.gensim_models as gensimvis\n",
    "pyLDAvis.enable_notebook()\n",
    "import re\n",
    "nlp = spacy.load(\"en_core_web_sm\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the translated csv file that needs to be analysed: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>review_idx</th>\n",
       "      <th>review_date</th>\n",
       "      <th>review_title</th>\n",
       "      <th>review_recommendation</th>\n",
       "      <th>review_rating</th>\n",
       "      <th>review_employee_info</th>\n",
       "      <th>Arbeitsatmosphäre_star</th>\n",
       "      <th>Arbeitsatmosphäre_plain_text</th>\n",
       "      <th>Work-Life-Balance_star</th>\n",
       "      <th>...</th>\n",
       "      <th>Spaßfaktor_star</th>\n",
       "      <th>Spaßfaktor_plain_text</th>\n",
       "      <th>Wie kann dich dein Arbeitgeber im Umgang mit der Corona-Situation noch besser unterstützen?_star</th>\n",
       "      <th>Wie kann dich dein Arbeitgeber im Umgang mit der Corona-Situation noch besser unterstützen?_plain_text</th>\n",
       "      <th>Wofür möchtest du deinen Arbeitgeber im Umgang mit der Corona-Situation loben?_star</th>\n",
       "      <th>Wofür möchtest du deinen Arbeitgeber im Umgang mit der Corona-Situation loben?_plain_text</th>\n",
       "      <th>Was macht dein Arbeitgeber im Umgang mit der Corona-Situation nicht gut?_star</th>\n",
       "      <th>Was macht dein Arbeitgeber im Umgang mit der Corona-Situation nicht gut?_plain_text</th>\n",
       "      <th>Wo siehst du Chancen für deinen Arbeitgeber mit der Corona-Situation besser umzugehen?_star</th>\n",
       "      <th>Wo siehst du Chancen für deinen Arbeitgeber mit der Corona-Situation besser umzugehen?_plain_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>review_0</td>\n",
       "      <td>2022-09-23T00:00:00+00:00</td>\n",
       "      <td>Utopian performance expectations, no cohesion,...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.8</td>\n",
       "      <td>Ex-employee Has worked in the field of IT at B...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>blasphemy, permanent dissatisfaction, pulling ...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>review_1</td>\n",
       "      <td>2022-09-23T00:00:00+00:00</td>\n",
       "      <td>Good employer with many freedoms.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.2</td>\n",
       "      <td>Employee Worked in IT at Bechtle Solingen in S...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>review_2</td>\n",
       "      <td>2022-09-21T00:00:00+00:00</td>\n",
       "      <td>Honest and fair employer</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.7</td>\n",
       "      <td>Employee Worked for Bechtle IT-Systemhaus Nure...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>The atmosphere is great, so 4 stars is always ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>review_3</td>\n",
       "      <td>2022-09-18T00:00:00+00:00</td>\n",
       "      <td>Even a red apple can be rotten inside</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.1</td>\n",
       "      <td>Employee Has worked in the field of logistics ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Very many employees are dissatisfied. No motiv...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>review_4</td>\n",
       "      <td>2022-09-12T00:00:00+00:00</td>\n",
       "      <td>Great company with a lot of potential and clea...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Manager / Management Worked at Bechtle GmbH &amp; ...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Working hours and places are flexible. A high ...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1641</th>\n",
       "      <td>1641</td>\n",
       "      <td>review_1641</td>\n",
       "      <td>2008-08-19T00:00:00+00:00</td>\n",
       "      <td>Very good: 4.08 out of 5 stars</td>\n",
       "      <td></td>\n",
       "      <td>4.3</td>\n",
       "      <td>Worked for Bechtle IT-Systemhaus Oberhausen in...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1642</th>\n",
       "      <td>1642</td>\n",
       "      <td>review_1642</td>\n",
       "      <td>2008-07-09T00:00:00+00:00</td>\n",
       "      <td>Good: 3.46 out of 5 stars</td>\n",
       "      <td></td>\n",
       "      <td>3.8</td>\n",
       "      <td>Ex-employee Has worked at Bechtle GmbH in Fran...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1643</th>\n",
       "      <td>1643</td>\n",
       "      <td>review_1643</td>\n",
       "      <td>2008-07-03T00:00:00+00:00</td>\n",
       "      <td>Sufficient: 1.62 out of 5 stars</td>\n",
       "      <td></td>\n",
       "      <td>1.5</td>\n",
       "      <td>Ex-employee Has worked at Bechtle GmbH in Fran...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1644</th>\n",
       "      <td>1644</td>\n",
       "      <td>review_1644</td>\n",
       "      <td>2008-01-30T00:00:00+00:00</td>\n",
       "      <td>Sufficient: 1.69 out of 5 stars</td>\n",
       "      <td></td>\n",
       "      <td>1.9</td>\n",
       "      <td>Ex-employee Has worked at Bechtle Köln GmbH in...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1645</th>\n",
       "      <td>1645</td>\n",
       "      <td>review_1645</td>\n",
       "      <td>2007-12-15T00:00:00+00:00</td>\n",
       "      <td>Satisfactory: 2.85 out of 5 stars</td>\n",
       "      <td></td>\n",
       "      <td>2.7</td>\n",
       "      <td>Ex-employee Has worked at Bechtle in Neckarsul...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1646 rows × 63 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Unnamed: 0   review_idx                review_date  \\\n",
       "0              0     review_0  2022-09-23T00:00:00+00:00   \n",
       "1              1     review_1  2022-09-23T00:00:00+00:00   \n",
       "2              2     review_2  2022-09-21T00:00:00+00:00   \n",
       "3              3     review_3  2022-09-18T00:00:00+00:00   \n",
       "4              4     review_4  2022-09-12T00:00:00+00:00   \n",
       "...          ...          ...                        ...   \n",
       "1641        1641  review_1641  2008-08-19T00:00:00+00:00   \n",
       "1642        1642  review_1642  2008-07-09T00:00:00+00:00   \n",
       "1643        1643  review_1643  2008-07-03T00:00:00+00:00   \n",
       "1644        1644  review_1644  2008-01-30T00:00:00+00:00   \n",
       "1645        1645  review_1645  2007-12-15T00:00:00+00:00   \n",
       "\n",
       "                                           review_title review_recommendation  \\\n",
       "0     Utopian performance expectations, no cohesion,...                   NaN   \n",
       "1                     Good employer with many freedoms.                   NaN   \n",
       "2                              Honest and fair employer                   NaN   \n",
       "3                 Even a red apple can be rotten inside                   NaN   \n",
       "4     Great company with a lot of potential and clea...                   NaN   \n",
       "...                                                 ...                   ...   \n",
       "1641                     Very good: 4.08 out of 5 stars                         \n",
       "1642                          Good: 3.46 out of 5 stars                         \n",
       "1643                    Sufficient: 1.62 out of 5 stars                         \n",
       "1644                    Sufficient: 1.69 out of 5 stars                         \n",
       "1645                  Satisfactory: 2.85 out of 5 stars                         \n",
       "\n",
       "      review_rating                               review_employee_info  \\\n",
       "0               2.8  Ex-employee Has worked in the field of IT at B...   \n",
       "1               4.2  Employee Worked in IT at Bechtle Solingen in S...   \n",
       "2               4.7  Employee Worked for Bechtle IT-Systemhaus Nure...   \n",
       "3               1.1  Employee Has worked in the field of logistics ...   \n",
       "4               5.0  Manager / Management Worked at Bechtle GmbH & ...   \n",
       "...             ...                                                ...   \n",
       "1641            4.3  Worked for Bechtle IT-Systemhaus Oberhausen in...   \n",
       "1642            3.8  Ex-employee Has worked at Bechtle GmbH in Fran...   \n",
       "1643            1.5  Ex-employee Has worked at Bechtle GmbH in Fran...   \n",
       "1644            1.9  Ex-employee Has worked at Bechtle Köln GmbH in...   \n",
       "1645            2.7  Ex-employee Has worked at Bechtle in Neckarsul...   \n",
       "\n",
       "      Arbeitsatmosphäre_star  \\\n",
       "0                        2.0   \n",
       "1                        NaN   \n",
       "2                        4.0   \n",
       "3                        1.0   \n",
       "4                        5.0   \n",
       "...                      ...   \n",
       "1641                     NaN   \n",
       "1642                     NaN   \n",
       "1643                     NaN   \n",
       "1644                     NaN   \n",
       "1645                     NaN   \n",
       "\n",
       "                           Arbeitsatmosphäre_plain_text  \\\n",
       "0     blasphemy, permanent dissatisfaction, pulling ...   \n",
       "1                                                         \n",
       "2     The atmosphere is great, so 4 stars is always ...   \n",
       "3     Very many employees are dissatisfied. No motiv...   \n",
       "4     Working hours and places are flexible. A high ...   \n",
       "...                                                 ...   \n",
       "1641                                                      \n",
       "1642                                                      \n",
       "1643                                                      \n",
       "1644                                                      \n",
       "1645                                                      \n",
       "\n",
       "      Work-Life-Balance_star  ... Spaßfaktor_star  Spaßfaktor_plain_text  \\\n",
       "0                        5.0  ...             NaN                          \n",
       "1                        NaN  ...             NaN                          \n",
       "2                        4.0  ...             NaN                          \n",
       "3                        NaN  ...             NaN                          \n",
       "4                        5.0  ...             NaN                          \n",
       "...                      ...  ...             ...                    ...   \n",
       "1641                     NaN  ...             NaN                          \n",
       "1642                     NaN  ...             NaN                          \n",
       "1643                     NaN  ...             NaN                          \n",
       "1644                     NaN  ...             NaN                          \n",
       "1645                     NaN  ...             NaN                          \n",
       "\n",
       "     Wie kann dich dein Arbeitgeber im Umgang mit der Corona-Situation noch besser unterstützen?_star  \\\n",
       "0                                                   NaN                                                 \n",
       "1                                                   NaN                                                 \n",
       "2                                                   NaN                                                 \n",
       "3                                                   NaN                                                 \n",
       "4                                                   NaN                                                 \n",
       "...                                                 ...                                                 \n",
       "1641                                                NaN                                                 \n",
       "1642                                                NaN                                                 \n",
       "1643                                                NaN                                                 \n",
       "1644                                                NaN                                                 \n",
       "1645                                                NaN                                                 \n",
       "\n",
       "      Wie kann dich dein Arbeitgeber im Umgang mit der Corona-Situation noch besser unterstützen?_plain_text  \\\n",
       "0                                                                                                              \n",
       "1                                                                                                              \n",
       "2                                                                                                              \n",
       "3                                                                                                              \n",
       "4                                                                                                              \n",
       "...                                                 ...                                                        \n",
       "1641                                                                                                           \n",
       "1642                                                                                                           \n",
       "1643                                                                                                           \n",
       "1644                                                                                                           \n",
       "1645                                                                                                           \n",
       "\n",
       "     Wofür möchtest du deinen Arbeitgeber im Umgang mit der Corona-Situation loben?_star  \\\n",
       "0                                                   NaN                                    \n",
       "1                                                   NaN                                    \n",
       "2                                                   NaN                                    \n",
       "3                                                   NaN                                    \n",
       "4                                                   NaN                                    \n",
       "...                                                 ...                                    \n",
       "1641                                                NaN                                    \n",
       "1642                                                NaN                                    \n",
       "1643                                                NaN                                    \n",
       "1644                                                NaN                                    \n",
       "1645                                                NaN                                    \n",
       "\n",
       "      Wofür möchtest du deinen Arbeitgeber im Umgang mit der Corona-Situation loben?_plain_text  \\\n",
       "0                                                                                                 \n",
       "1                                                                                                 \n",
       "2                                                                                                 \n",
       "3                                                                                                 \n",
       "4                                                                                                 \n",
       "...                                                 ...                                           \n",
       "1641                                                                                              \n",
       "1642                                                                                              \n",
       "1643                                                                                              \n",
       "1644                                                                                              \n",
       "1645                                                                                              \n",
       "\n",
       "     Was macht dein Arbeitgeber im Umgang mit der Corona-Situation nicht gut?_star  \\\n",
       "0                                                   NaN                              \n",
       "1                                                   NaN                              \n",
       "2                                                   NaN                              \n",
       "3                                                   NaN                              \n",
       "4                                                   NaN                              \n",
       "...                                                 ...                              \n",
       "1641                                                NaN                              \n",
       "1642                                                NaN                              \n",
       "1643                                                NaN                              \n",
       "1644                                                NaN                              \n",
       "1645                                                NaN                              \n",
       "\n",
       "      Was macht dein Arbeitgeber im Umgang mit der Corona-Situation nicht gut?_plain_text  \\\n",
       "0                                                                                           \n",
       "1                                                                                           \n",
       "2                                                                                           \n",
       "3                                                                                           \n",
       "4                                                                                           \n",
       "...                                                 ...                                     \n",
       "1641                                                                                        \n",
       "1642                                                                                        \n",
       "1643                                                                                        \n",
       "1644                                                                                        \n",
       "1645                                                                                        \n",
       "\n",
       "     Wo siehst du Chancen für deinen Arbeitgeber mit der Corona-Situation besser umzugehen?_star  \\\n",
       "0                                                   NaN                                            \n",
       "1                                                   NaN                                            \n",
       "2                                                   NaN                                            \n",
       "3                                                   NaN                                            \n",
       "4                                                   NaN                                            \n",
       "...                                                 ...                                            \n",
       "1641                                                NaN                                            \n",
       "1642                                                NaN                                            \n",
       "1643                                                NaN                                            \n",
       "1644                                                NaN                                            \n",
       "1645                                                NaN                                            \n",
       "\n",
       "      Wo siehst du Chancen für deinen Arbeitgeber mit der Corona-Situation besser umzugehen?_plain_text  \n",
       "0                                                                                                        \n",
       "1                                                                                                        \n",
       "2                                                                                                        \n",
       "3                                                                                                        \n",
       "4                                                                                                        \n",
       "...                                                 ...                                                  \n",
       "1641                                                                                                     \n",
       "1642                                                                                                     \n",
       "1643                                                                                                     \n",
       "1644                                                                                                     \n",
       "1645                                                                                                     \n",
       "\n",
       "[1646 rows x 63 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here, we load in Bechtle's translated dataframe and store it into input_df: \n",
    "# The translated Bechtle file can be found in \"translated_csvs_folder\" folder/directory. \n",
    "input_df = pd.read_csv('bechtle_translated.csv',sep='\\t', encoding= 'utf-8')\n",
    "input_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def construct_time(df): \n",
    "    '''\n",
    "    Purpose: The purpose of the function is to extract the year and month that will be further utilised in our analysis. The year and \n",
    "    month is extracted from the review_date column and placed into new columns, namely, year and month. \n",
    "    \n",
    "    Parameters:\n",
    "        df: Takes input the input_df that has been loaded above. \n",
    "    Return: Returns the dataframe with the newly extracted year and month columns. \n",
    "    '''\n",
    "    \n",
    "    df.loc[:,\"review_date\"] = pd.to_datetime(df.loc[:,\"review_date\"] )\n",
    "    df.insert(2, \"year\", df[\"review_date\"].dt.year)\n",
    "    df.insert(3, \"month\", df[\"review_date\"].dt.month)\n",
    "    df.pop(\"review_date\")\n",
    "\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>review_idx</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>review_title</th>\n",
       "      <th>review_recommendation</th>\n",
       "      <th>review_rating</th>\n",
       "      <th>review_employee_info</th>\n",
       "      <th>Arbeitsatmosphäre_star</th>\n",
       "      <th>Arbeitsatmosphäre_plain_text</th>\n",
       "      <th>...</th>\n",
       "      <th>Spaßfaktor_star</th>\n",
       "      <th>Spaßfaktor_plain_text</th>\n",
       "      <th>Wie kann dich dein Arbeitgeber im Umgang mit der Corona-Situation noch besser unterstützen?_star</th>\n",
       "      <th>Wie kann dich dein Arbeitgeber im Umgang mit der Corona-Situation noch besser unterstützen?_plain_text</th>\n",
       "      <th>Wofür möchtest du deinen Arbeitgeber im Umgang mit der Corona-Situation loben?_star</th>\n",
       "      <th>Wofür möchtest du deinen Arbeitgeber im Umgang mit der Corona-Situation loben?_plain_text</th>\n",
       "      <th>Was macht dein Arbeitgeber im Umgang mit der Corona-Situation nicht gut?_star</th>\n",
       "      <th>Was macht dein Arbeitgeber im Umgang mit der Corona-Situation nicht gut?_plain_text</th>\n",
       "      <th>Wo siehst du Chancen für deinen Arbeitgeber mit der Corona-Situation besser umzugehen?_star</th>\n",
       "      <th>Wo siehst du Chancen für deinen Arbeitgeber mit der Corona-Situation besser umzugehen?_plain_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>review_0</td>\n",
       "      <td>2022</td>\n",
       "      <td>9</td>\n",
       "      <td>Utopian performance expectations, no cohesion,...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.8</td>\n",
       "      <td>Ex-employee Has worked in the field of IT at B...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>blasphemy, permanent dissatisfaction, pulling ...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>review_1</td>\n",
       "      <td>2022</td>\n",
       "      <td>9</td>\n",
       "      <td>Good employer with many freedoms.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.2</td>\n",
       "      <td>Employee Worked in IT at Bechtle Solingen in S...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>review_2</td>\n",
       "      <td>2022</td>\n",
       "      <td>9</td>\n",
       "      <td>Honest and fair employer</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.7</td>\n",
       "      <td>Employee Worked for Bechtle IT-Systemhaus Nure...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>The atmosphere is great, so 4 stars is always ...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>review_3</td>\n",
       "      <td>2022</td>\n",
       "      <td>9</td>\n",
       "      <td>Even a red apple can be rotten inside</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.1</td>\n",
       "      <td>Employee Has worked in the field of logistics ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Very many employees are dissatisfied. No motiv...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>review_4</td>\n",
       "      <td>2022</td>\n",
       "      <td>9</td>\n",
       "      <td>Great company with a lot of potential and clea...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Manager / Management Worked at Bechtle GmbH &amp; ...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Working hours and places are flexible. A high ...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1641</th>\n",
       "      <td>1641</td>\n",
       "      <td>review_1641</td>\n",
       "      <td>2008</td>\n",
       "      <td>8</td>\n",
       "      <td>Very good: 4.08 out of 5 stars</td>\n",
       "      <td></td>\n",
       "      <td>4.3</td>\n",
       "      <td>Worked for Bechtle IT-Systemhaus Oberhausen in...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1642</th>\n",
       "      <td>1642</td>\n",
       "      <td>review_1642</td>\n",
       "      <td>2008</td>\n",
       "      <td>7</td>\n",
       "      <td>Good: 3.46 out of 5 stars</td>\n",
       "      <td></td>\n",
       "      <td>3.8</td>\n",
       "      <td>Ex-employee Has worked at Bechtle GmbH in Fran...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1643</th>\n",
       "      <td>1643</td>\n",
       "      <td>review_1643</td>\n",
       "      <td>2008</td>\n",
       "      <td>7</td>\n",
       "      <td>Sufficient: 1.62 out of 5 stars</td>\n",
       "      <td></td>\n",
       "      <td>1.5</td>\n",
       "      <td>Ex-employee Has worked at Bechtle GmbH in Fran...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1644</th>\n",
       "      <td>1644</td>\n",
       "      <td>review_1644</td>\n",
       "      <td>2008</td>\n",
       "      <td>1</td>\n",
       "      <td>Sufficient: 1.69 out of 5 stars</td>\n",
       "      <td></td>\n",
       "      <td>1.9</td>\n",
       "      <td>Ex-employee Has worked at Bechtle Köln GmbH in...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1645</th>\n",
       "      <td>1645</td>\n",
       "      <td>review_1645</td>\n",
       "      <td>2007</td>\n",
       "      <td>12</td>\n",
       "      <td>Satisfactory: 2.85 out of 5 stars</td>\n",
       "      <td></td>\n",
       "      <td>2.7</td>\n",
       "      <td>Ex-employee Has worked at Bechtle in Neckarsul...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1646 rows × 64 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Unnamed: 0   review_idx  year  month  \\\n",
       "0              0     review_0  2022      9   \n",
       "1              1     review_1  2022      9   \n",
       "2              2     review_2  2022      9   \n",
       "3              3     review_3  2022      9   \n",
       "4              4     review_4  2022      9   \n",
       "...          ...          ...   ...    ...   \n",
       "1641        1641  review_1641  2008      8   \n",
       "1642        1642  review_1642  2008      7   \n",
       "1643        1643  review_1643  2008      7   \n",
       "1644        1644  review_1644  2008      1   \n",
       "1645        1645  review_1645  2007     12   \n",
       "\n",
       "                                           review_title review_recommendation  \\\n",
       "0     Utopian performance expectations, no cohesion,...                   NaN   \n",
       "1                     Good employer with many freedoms.                   NaN   \n",
       "2                              Honest and fair employer                   NaN   \n",
       "3                 Even a red apple can be rotten inside                   NaN   \n",
       "4     Great company with a lot of potential and clea...                   NaN   \n",
       "...                                                 ...                   ...   \n",
       "1641                     Very good: 4.08 out of 5 stars                         \n",
       "1642                          Good: 3.46 out of 5 stars                         \n",
       "1643                    Sufficient: 1.62 out of 5 stars                         \n",
       "1644                    Sufficient: 1.69 out of 5 stars                         \n",
       "1645                  Satisfactory: 2.85 out of 5 stars                         \n",
       "\n",
       "      review_rating                               review_employee_info  \\\n",
       "0               2.8  Ex-employee Has worked in the field of IT at B...   \n",
       "1               4.2  Employee Worked in IT at Bechtle Solingen in S...   \n",
       "2               4.7  Employee Worked for Bechtle IT-Systemhaus Nure...   \n",
       "3               1.1  Employee Has worked in the field of logistics ...   \n",
       "4               5.0  Manager / Management Worked at Bechtle GmbH & ...   \n",
       "...             ...                                                ...   \n",
       "1641            4.3  Worked for Bechtle IT-Systemhaus Oberhausen in...   \n",
       "1642            3.8  Ex-employee Has worked at Bechtle GmbH in Fran...   \n",
       "1643            1.5  Ex-employee Has worked at Bechtle GmbH in Fran...   \n",
       "1644            1.9  Ex-employee Has worked at Bechtle Köln GmbH in...   \n",
       "1645            2.7  Ex-employee Has worked at Bechtle in Neckarsul...   \n",
       "\n",
       "      Arbeitsatmosphäre_star  \\\n",
       "0                        2.0   \n",
       "1                        NaN   \n",
       "2                        4.0   \n",
       "3                        1.0   \n",
       "4                        5.0   \n",
       "...                      ...   \n",
       "1641                     NaN   \n",
       "1642                     NaN   \n",
       "1643                     NaN   \n",
       "1644                     NaN   \n",
       "1645                     NaN   \n",
       "\n",
       "                           Arbeitsatmosphäre_plain_text  ...  Spaßfaktor_star  \\\n",
       "0     blasphemy, permanent dissatisfaction, pulling ...  ...              NaN   \n",
       "1                                                        ...              NaN   \n",
       "2     The atmosphere is great, so 4 stars is always ...  ...              NaN   \n",
       "3     Very many employees are dissatisfied. No motiv...  ...              NaN   \n",
       "4     Working hours and places are flexible. A high ...  ...              NaN   \n",
       "...                                                 ...  ...              ...   \n",
       "1641                                                     ...              NaN   \n",
       "1642                                                     ...              NaN   \n",
       "1643                                                     ...              NaN   \n",
       "1644                                                     ...              NaN   \n",
       "1645                                                     ...              NaN   \n",
       "\n",
       "     Spaßfaktor_plain_text  \\\n",
       "0                            \n",
       "1                            \n",
       "2                            \n",
       "3                            \n",
       "4                            \n",
       "...                    ...   \n",
       "1641                         \n",
       "1642                         \n",
       "1643                         \n",
       "1644                         \n",
       "1645                         \n",
       "\n",
       "      Wie kann dich dein Arbeitgeber im Umgang mit der Corona-Situation noch besser unterstützen?_star  \\\n",
       "0                                                   NaN                                                  \n",
       "1                                                   NaN                                                  \n",
       "2                                                   NaN                                                  \n",
       "3                                                   NaN                                                  \n",
       "4                                                   NaN                                                  \n",
       "...                                                 ...                                                  \n",
       "1641                                                NaN                                                  \n",
       "1642                                                NaN                                                  \n",
       "1643                                                NaN                                                  \n",
       "1644                                                NaN                                                  \n",
       "1645                                                NaN                                                  \n",
       "\n",
       "     Wie kann dich dein Arbeitgeber im Umgang mit der Corona-Situation noch besser unterstützen?_plain_text  \\\n",
       "0                                                                                                             \n",
       "1                                                                                                             \n",
       "2                                                                                                             \n",
       "3                                                                                                             \n",
       "4                                                                                                             \n",
       "...                                                 ...                                                       \n",
       "1641                                                                                                          \n",
       "1642                                                                                                          \n",
       "1643                                                                                                          \n",
       "1644                                                                                                          \n",
       "1645                                                                                                          \n",
       "\n",
       "      Wofür möchtest du deinen Arbeitgeber im Umgang mit der Corona-Situation loben?_star  \\\n",
       "0                                                   NaN                                     \n",
       "1                                                   NaN                                     \n",
       "2                                                   NaN                                     \n",
       "3                                                   NaN                                     \n",
       "4                                                   NaN                                     \n",
       "...                                                 ...                                     \n",
       "1641                                                NaN                                     \n",
       "1642                                                NaN                                     \n",
       "1643                                                NaN                                     \n",
       "1644                                                NaN                                     \n",
       "1645                                                NaN                                     \n",
       "\n",
       "     Wofür möchtest du deinen Arbeitgeber im Umgang mit der Corona-Situation loben?_plain_text  \\\n",
       "0                                                                                                \n",
       "1                                                                                                \n",
       "2                                                                                                \n",
       "3                                                                                                \n",
       "4                                                                                                \n",
       "...                                                 ...                                          \n",
       "1641                                                                                             \n",
       "1642                                                                                             \n",
       "1643                                                                                             \n",
       "1644                                                                                             \n",
       "1645                                                                                             \n",
       "\n",
       "      Was macht dein Arbeitgeber im Umgang mit der Corona-Situation nicht gut?_star  \\\n",
       "0                                                   NaN                               \n",
       "1                                                   NaN                               \n",
       "2                                                   NaN                               \n",
       "3                                                   NaN                               \n",
       "4                                                   NaN                               \n",
       "...                                                 ...                               \n",
       "1641                                                NaN                               \n",
       "1642                                                NaN                               \n",
       "1643                                                NaN                               \n",
       "1644                                                NaN                               \n",
       "1645                                                NaN                               \n",
       "\n",
       "     Was macht dein Arbeitgeber im Umgang mit der Corona-Situation nicht gut?_plain_text  \\\n",
       "0                                                                                          \n",
       "1                                                                                          \n",
       "2                                                                                          \n",
       "3                                                                                          \n",
       "4                                                                                          \n",
       "...                                                 ...                                    \n",
       "1641                                                                                       \n",
       "1642                                                                                       \n",
       "1643                                                                                       \n",
       "1644                                                                                       \n",
       "1645                                                                                       \n",
       "\n",
       "      Wo siehst du Chancen für deinen Arbeitgeber mit der Corona-Situation besser umzugehen?_star  \\\n",
       "0                                                   NaN                                             \n",
       "1                                                   NaN                                             \n",
       "2                                                   NaN                                             \n",
       "3                                                   NaN                                             \n",
       "4                                                   NaN                                             \n",
       "...                                                 ...                                             \n",
       "1641                                                NaN                                             \n",
       "1642                                                NaN                                             \n",
       "1643                                                NaN                                             \n",
       "1644                                                NaN                                             \n",
       "1645                                                NaN                                             \n",
       "\n",
       "     Wo siehst du Chancen für deinen Arbeitgeber mit der Corona-Situation besser umzugehen?_plain_text  \n",
       "0                                                                                                       \n",
       "1                                                                                                       \n",
       "2                                                                                                       \n",
       "3                                                                                                       \n",
       "4                                                                                                       \n",
       "...                                                 ...                                                 \n",
       "1641                                                                                                    \n",
       "1642                                                                                                    \n",
       "1643                                                                                                    \n",
       "1644                                                                                                    \n",
       "1645                                                                                                    \n",
       "\n",
       "[1646 rows x 64 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# As you can see, the input_df_time will contain the year and month columns for Bechtle's translated dataframe. \n",
    "input_df_time = construct_time(input_df)\n",
    "input_df_time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim\n",
    "from gensim.utils import simple_preprocess \n",
    "from nltk.corpus import stopwords\n",
    "# stopwords are loaded in order to remove a corpus of stopwords that add no weight to our topic extraction\n",
    "stopwords = stopwords.words('english')\n",
    "import spacy\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "from pprint import pprint\n",
    "\n",
    "\n",
    "def topic_modelling(df, year:None, topic_col):\n",
    "    \n",
    "    '''\n",
    "    Purpose: The motivation behind this function is to extract analyse words and cluster them in order to identify key topics that have been spoken\n",
    "    about in the reviews. \n",
    "    df: The dataframe to use for topic extraction \n",
    "    year: The year for which we will perform topic modelling\n",
    "    topic_col: The column for which we will perform topic modelling(str)\n",
    "    \n",
    "    Returns: Returns a dashboard of topics, where words have been clustered. The next step would be to utilise these clustered words on a yearly basis\n",
    "    and analyze the topics that can be generated from them. The LDA model utilized is an approximation of key topics. \n",
    "    '''\n",
    "    df= df[[topic_col, 'year']]\n",
    "    df= df.dropna(how='any', axis=0)\n",
    "    df['len']= df[topic_col].map(lambda x: len(x))\n",
    "    df = df.drop(df[df.len == 1].index)\n",
    "\n",
    "    def sent_to_words(sentences):\n",
    "        '''\n",
    "        Purpose: The purpose of the function is to take input th review and create a word list for the same review. Operation is performed \n",
    "        row wise in the later stages. \n",
    "        Paramters: \n",
    "            sentences: Conversion of a list of words into tokens.\n",
    "        Returns: Tokenized reviews \n",
    "        '''\n",
    "        for sentence in sentences:\n",
    "            yield(gensim.utils.simple_preprocess(str(sentence), deacc=True))       \n",
    "\n",
    "    def eliminate_stopwords(data): \n",
    "        '''Purpose: The purpose of this function is to elimate any stopwords in the review and further curate the review text. Operation is \n",
    "        performed row-wise. \n",
    "        \n",
    "        Parameters: \n",
    "            data: Takes input each row(review)\n",
    "        \n",
    "        Return: Returns the review after the removal of stopwords\n",
    "        '''\n",
    "        doc= []\n",
    "        for word_list in data: \n",
    "            temp=[]\n",
    "            for word in word_list: \n",
    "                if word not in stopwords:\n",
    "                    temp.append(word)\n",
    "                else: \n",
    "                    continue\n",
    "            doc.append(temp)\n",
    "        return doc\n",
    "\n",
    "    def make_trigram(texts):\n",
    "        '''Purpose: The purpose of the function is to look into n-grams. Here n-grams is to consider phrases. Trigram considers phrases such\n",
    "        as \"flexible working hours\". Since the phrase conveys more information than just the word \"flexible\n",
    "        Parameters: \n",
    "            texts: Takes input row after the stopwords have been eliminated \n",
    "        Returns: Any trigram/ phrases that convey meaning. \n",
    "        \"'''\n",
    "        return [trigram_mod[bigram_mod[doc]] for doc in texts]\n",
    "    def lemmatization(texts, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV']):\n",
    "        '''Purpose: The purpose of the function si to identify parts-ofspeeches. For instance here, we allow- nouns, adjectives,verbs and adverbs. \n",
    "        Therefore, we will consider these words and extract topics based out of these words for our analysis. \n",
    "        Paramterers: \n",
    "            texts: Takes input the review, the operation is performed row-wise as well. \n",
    "        Returns: Return the nouns, adjectives, verbs and adverbs found in the review. \n",
    "        '''\n",
    "        texts_out = []\n",
    "        for sent in texts:\n",
    "            doc = nlp(\" \".join(sent)) \n",
    "            texts_out.append([token.lemma_ for token in doc\n",
    "                             if token.pos_ in allowed_postags])\n",
    "            \n",
    "        return texts_out\n",
    "\n",
    "    if year is None:\n",
    "        \n",
    "\n",
    "        topic_df= df[[topic_col]]\n",
    "        topic_df= topic_df.dropna(how='any', axis=0)\n",
    "        topic_df['len']= topic_df[topic_col].map(lambda x: len(x))\n",
    "        topic_df = topic_df.drop(topic_df[topic_df.len == 1].index) # dropping text rows with no text\n",
    "        data = topic_df[topic_col].values.tolist() # convert to list\n",
    "        data = [re.sub(r'[^a-zA-Z ]+', '', str(sent)) for sent in data] # removing special chracters\n",
    "\n",
    "\n",
    "        data_words = list(sent_to_words(data))\n",
    "        no_stopwords = eliminate_stopwords(data_words)\n",
    "            # # Creating and Applying Bigrams and Trigrams\n",
    "        bigram = gensim.models.Phrases(data_words, min_count=2, threshold=30)\n",
    "        trigram = gensim.models.Phrases(bigram[data_words],min_count= 3, threshold=30)\n",
    "        bigram_mod = gensim.models.phrases.Phraser(bigram)\n",
    "        trigram_mod = gensim.models.phrases.Phraser(trigram)\n",
    "        data_words_trigrams = make_trigram(no_stopwords)\n",
    "\n",
    "        data_lemmatized = lemmatization(data_words_trigrams,\n",
    "                                    allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV'])\n",
    "\n",
    "        corpora_dict = corpora.Dictionary(data_lemmatized)      \n",
    "        texts = data_lemmatized                       \n",
    "        corpus = [corpora_dict.doc2bow(text) for text in texts] \n",
    "\n",
    "\n",
    "        #LDA Model\n",
    "        lda_model = gensim.models.ldamodel.LdaModel\\\n",
    "                    (corpus=corpus, id2word=corpora_dict, num_topics =8, random_state = 42,\n",
    "                     update_every = 1, chunksize = 50, passes = 5, alpha = 'auto',\n",
    "                     per_word_topics=True) \n",
    "        pprint(lda_model.print_topics())\n",
    "        doc_lda = lda_model[corpus]\n",
    "        pyLDAvis.enable_notebook()\n",
    "        vis = gensimvis.prepare(lda_model, corpus, corpora_dict)\n",
    "        return vis \n",
    "    \n",
    "    else: \n",
    "        print(year)\n",
    "        time_df =  df[df['year']==year]\n",
    "        topic_df= time_df[[topic_col]]\n",
    "        topic_df= topic_df.dropna(how='any', axis=0)\n",
    "        topic_df['len']= topic_df[topic_col].map(lambda x: len(x))\n",
    "        topic_df = topic_df.drop(topic_df[topic_df.len == 1].index) # dropping text rows with no text\n",
    "        data = topic_df[topic_col].values.tolist() # convert to list\n",
    "        data = [re.sub(r'[^a-zA-Z ]+', '', str(sent)) for sent in data] # removing special chracters\n",
    "\n",
    "\n",
    "        data_words = list(sent_to_words(data))\n",
    "        no_stopwords = eliminate_stopwords(data_words)\n",
    "            # # Create and Apply Bigrams and Trigrams\n",
    "        bigram = gensim.models.Phrases(data_words, min_count=2, threshold=30)\n",
    "        # # Higher threshold fewer phrases\n",
    "        trigram = gensim.models.Phrases(bigram[data_words],min_count= 3, threshold=30)\n",
    "        bigram_mod = gensim.models.phrases.Phraser(bigram)\n",
    "        # # Faster way to get a sentence into a trigram/bigram\n",
    "        trigram_mod = gensim.models.phrases.Phraser(trigram)\n",
    "        data_words_trigrams = make_trigram(no_stopwords)\n",
    "\n",
    "        data_lemmatized = lemmatization(data_words_trigrams,\n",
    "                                    allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV'])\n",
    "\n",
    "        corpora_dict = corpora.Dictionary(data_lemmatized)      \n",
    "        texts = data_lemmatized                       \n",
    "        corpus = [corpora_dict.doc2bow(text) for text in texts] \n",
    "\n",
    "\n",
    "        # Building the LDA Model\n",
    "        lda_model = gensim.models.ldamodel.LdaModel\\\n",
    "                    (corpus=corpus, id2word=corpora_dict, num_topics =8, random_state = 42,\n",
    "                     update_every = 1, chunksize = 50, passes = 5, alpha = 'auto',\n",
    "                     per_word_topics=True) \n",
    "        pprint(lda_model.print_topics())\n",
    "        doc_lda = lda_model[corpus]\n",
    "        pyLDAvis.enable_notebook()\n",
    "        vis = gensimvis.prepare(lda_model, corpus, corpora_dict)\n",
    "        # Compute perplexity\n",
    "        perplexlity = lda_model.log_perplexity(corpus)\n",
    "        return vis \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2016\n",
      "[(0,\n",
      "  '0.062*\"work\" + 0.060*\"great\" + 0.052*\"climate\" + 0.039*\"good\" + '\n",
      "  '0.031*\"salary\" + 0.031*\"contract\" + 0.031*\"young\" + 0.031*\"working_hour\" + '\n",
      "  '0.027*\"colleague\" + 0.019*\"flexible\"'),\n",
      " (1,\n",
      "  '0.026*\"level\" + 0.025*\"personal\" + 0.016*\"flat\" + 0.015*\"hierarchy\" + '\n",
      "  '0.014*\"mutual\" + 0.014*\"hire\" + 0.014*\"orientation\" + 0.014*\"fire\" + '\n",
      "  '0.014*\"mentality\" + 0.014*\"longterm\"'),\n",
      " (2,\n",
      "  '0.100*\"colleague\" + 0.032*\"good\" + 0.025*\"cohesion\" + 0.020*\"short\" + '\n",
      "  '0.020*\"top\" + 0.020*\"brand\" + 0.015*\"perhaps\" + 0.015*\"commitment\" + '\n",
      "  '0.015*\"constructively\" + 0.015*\"superior\"'),\n",
      " (3,\n",
      "  '0.066*\"work\" + 0.060*\"good\" + 0.058*\"team\" + 0.046*\"atmosphere\" + '\n",
      "  '0.028*\"building\" + 0.019*\"time\" + 0.019*\"level\" + 0.019*\"reputation\" + '\n",
      "  '0.019*\"office\" + 0.019*\"nice\"'),\n",
      " (4,\n",
      "  '0.077*\"training\" + 0.067*\"opportunity\" + 0.036*\"company\" + 0.034*\"offer\" + '\n",
      "  '0.031*\"employee\" + 0.027*\"good\" + 0.020*\"give\" + 0.020*\"management\" + '\n",
      "  '0.015*\"train\" + 0.015*\"employment\"'),\n",
      " (5,\n",
      "  '0.036*\"achieve\" + 0.025*\"thing\" + 0.025*\"group\" + 0.024*\"flexible\" + '\n",
      "  '0.019*\"sell\" + 0.019*\"hardware\" + 0.019*\"slogan\" + 0.019*\"right\" + '\n",
      "  '0.019*\"attention\" + 0.019*\"futureproof\"'),\n",
      " (6,\n",
      "  '0.050*\"good\" + 0.042*\"opportunity\" + 0.035*\"new\" + 0.035*\"company\" + '\n",
      "  '0.032*\"development\" + 0.032*\"great\" + 0.028*\"colleague\" + 0.025*\"open\" + '\n",
      "  '0.025*\"find\" + 0.025*\"successful\"'),\n",
      " (7,\n",
      "  '0.092*\"also\" + 0.039*\"individual\" + 0.034*\"employee\" + 0.028*\"make\" + '\n",
      "  '0.028*\"possibility\" + 0.023*\"positive\" + 0.023*\"goal\" + 0.023*\"environment\" '\n",
      "  '+ 0.018*\"well\" + 0.012*\"part\"')]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\karti\\anaconda3\\lib\\site-packages\\pyLDAvis\\_prepare.py:246: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only.\n",
      "  default_term_info = default_term_info.sort_values(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v1.0.0.css\">\n",
       "\n",
       "\n",
       "<div id=\"ldavis_el3072424679739468324926357982\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "\n",
       "var ldavis_el3072424679739468324926357982_data = {\"mdsDat\": {\"x\": [-0.09149268528007641, -0.11147324315910767, 0.04525119256511706, 0.11733138283854672, -0.09793501198966063, -0.022624992017168017, 0.13798935399942241, 0.02295400304292686], \"y\": [0.10399558599698358, -0.013936584026318179, 0.12582116034902407, 0.04095292819515133, -0.024012923494041834, -0.15701673373890682, -0.05825651033130582, -0.017546922950586363], \"topics\": [1, 2, 3, 4, 5, 6, 7, 8], \"cluster\": [1, 1, 1, 1, 1, 1, 1, 1], \"Freq\": [22.424871858885993, 17.0797290676498, 14.076797334768074, 13.82069193049506, 12.95037606469917, 9.469733520483626, 6.975067829075994, 3.2027323939422816]}, \"tinfo\": {\"Term\": [\"also\", \"colleague\", \"work\", \"training\", \"opportunity\", \"great\", \"team\", \"climate\", \"atmosphere\", \"good\", \"individual\", \"new\", \"employee\", \"offer\", \"level\", \"development\", \"achieve\", \"salary\", \"contract\", \"young\", \"flexible\", \"company\", \"working_hour\", \"make\", \"building\", \"possibility\", \"open\", \"find\", \"successful\", \"employment\", \"new\", \"development\", \"open\", \"find\", \"successful\", \"market\", \"supervisor\", \"vary\", \"task\", \"chance\", \"usually\", \"perspective\", \"communication\", \"relax\", \"arebite\", \"strong\", \"future\", \"always\", \"constant\", \"interesting\", \"many\", \"sustainable\", \"flexibly\", \"financially\", \"organize\", \"positioning\", \"help\", \"mood\", \"cheerful\", \"futureoriente\", \"company\", \"opportunity\", \"great\", \"good\", \"colleague\", \"top\", \"employer\", \"team\", \"bechtle\", \"salary\", \"contract\", \"young\", \"climate\", \"innovative\", \"advancement\", \"working_hour\", \"teamwork\", \"reasonable\", \"openness\", \"friendly\", \"great\", \"flexible\", \"work\", \"slightly\", \"performance\", \"subsidize\", \"food\", \"restaurant\", \"base\", \"compensate\", \"average\", \"especially\", \"strength\", \"service\", \"portfolio\", \"basis\", \"economic\", \"stable\", \"pioneering\", \"super\", \"good\", \"colleague\", \"interesting\", \"opportunity\", \"working\", \"company\", \"employee\", \"manager\", \"flat\", \"nice\", \"house\", \"system\", \"excellent\", \"hierarchy\", \"live\", \"training\", \"offer\", \"give\", \"management\", \"develop\", \"brange\", \"default\", \"fixedterm\", \"customer\", \"resource\", \"employment\", \"train\", \"feel\", \"alone\", \"leave\", \"take\", \"wellknown\", \"htle\", \"evolution\", \"share\", \"advantage\", \"recognition\", \"various\", \"leader\", \"encouragement\", \"measure\", \"regionally\", \"interact\", \"due\", \"centrally\", \"opportunity\", \"internal\", \"employer\", \"safe\", \"employee\", \"responsibility\", \"company\", \"bechtle\", \"good\", \"large\", \"also\", \"individual\", \"make\", \"positive\", \"goal\", \"environment\", \"possibility\", \"part\", \"trust\", \"quick\", \"regard\", \"contribution\", \"promotion\", \"associate\", \"brilliant\", \"skill\", \"transfer\", \"structured\", \"realise\", \"switch\", \"little\", \"retrospect\", \"simply\", \"decision\", \"decentralise\", \"activity\", \"concept\", \"corporation\", \"close\", \"act\", \"well\", \"employee\", \"company\", \"large\", \"responsibility\", \"area\", \"atmosphere\", \"building\", \"time\", \"reputation\", \"office\", \"place\", \"humanly\", \"specialist\", \"arrive\", \"much\", \"though\", \"harmonious\", \"small\", \"accept\", \"improvement\", \"stateoftheart\", \"suggestion\", \"helpful\", \"familiar\", \"team\", \"nice\", \"level\", \"collaboration\", \"impeccable\", \"ready\", \"home\", \"private\", \"important\", \"approve\", \"appointment\", \"work\", \"safe\", \"good\", \"many\", \"free\", \"cohesion\", \"short\", \"achievement\", \"acknowledge\", \"constructively\", \"criticism\", \"express\", \"openly\", \"praise\", \"superior\", \"coherence\", \"breakfast\", \"decisionmake\", \"design\", \"option\", \"path\", \"position\", \"pay\", \"promote\", \"canteen\", \"hard\", \"likely\", \"actually\", \"compact\", \"personality\", \"promise\", \"unusual\", \"perhaps\", \"brand\", \"colleague\", \"top\", \"good\", \"commitment\", \"train\", \"super\", \"achieve\", \"thing\", \"group\", \"hardware\", \"sell\", \"attention\", \"dynamic\", \"futureproof\", \"right\", \"slogan\", \"openende\", \"outset\", \"door\", \"effort\", \"really\", \"show\", \"appreciation\", \"capitalise\", \"feedback\", \"objective\", \"person\", \"regular\", \"enormous\", \"commitment\", \"still\", \"solid\", \"happy\", \"overlook\", \"certain\", \"entrepreneur\", \"employment\", \"flexible\", \"company\", \"large\", \"personal\", \"hire\", \"mutual\", \"orientation\", \"fire\", \"mentality\", \"longterm\", \"helpfulness\", \"flat\", \"level\", \"hierarchy\", \"responsibility\", \"high\", \"atmosphere\", \"solid\", \"still\", \"happy\", \"overlook\", \"certain\", \"entrepreneur\", \"become\", \"type\", \"family\", \"unfortunately\", \"say\", \"bad\", \"approach\", \"flexibility\", \"organisation\", \"internally\", \"colleague\", \"academy\", \"available\", \"explain\", \"structure\", \"decentralize\", \"use\", \"scene\", \"know\", \"theoretically\", \"eye\", \"centrally\", \"due\", \"interact\", \"regionally\", \"measure\", \"well\", \"resource\", \"bechtle\", \"employee\", \"many\", \"strong\", \"internal\"], \"Freq\": [8.0, 13.0, 12.0, 7.0, 12.0, 10.0, 7.0, 6.0, 4.0, 19.0, 3.0, 5.0, 7.0, 3.0, 2.0, 4.0, 2.0, 3.0, 3.0, 3.0, 3.0, 10.0, 4.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 2.0, 4.7965688970557405, 4.387454225521339, 3.399293352871316, 3.39854683192365, 3.3974677950553946, 1.999854941473529, 1.9998318570981617, 1.9997880850858536, 1.9997880850858536, 1.9991007257340738, 1.9955799169861346, 1.5032875736711204, 1.4984732830378538, 1.4984732830378538, 1.4942720528651579, 2.085347105708988, 2.082276000776261, 1.0930642213984574, 1.0917076674492854, 1.7788897867295108, 1.9893509199608312, 0.5995486584772388, 0.5995486584772388, 0.5995486584772388, 0.5995485954051749, 0.5995485954051749, 0.5995485954051749, 0.5995485323331111, 0.5995485323331111, 0.5995485323331111, 4.698631605470699, 5.699539846800248, 4.342282013395759, 6.815912853969029, 3.7464061673403255, 1.0948491608052626, 1.1010504061224742, 1.9891171748922227, 1.0948297346095983, 3.1586058229153777, 3.157678874983353, 3.157293415291885, 5.3186320061171735, 1.8567324323270094, 1.8563825210118732, 3.156507123638103, 1.3971165219903494, 1.3799121646893848, 1.0173123850434271, 1.0173060439817032, 6.186870161326008, 1.9319101385934738, 6.361298165697281, 0.5571120882228805, 0.5571119921461877, 0.5571119921461877, 0.5571119921461877, 0.5571119921461877, 0.5571119441078413, 0.5571119441078413, 0.5571118960694948, 0.5571118480311485, 0.5571112235326454, 0.557111175494299, 0.5571111274559526, 0.5571109833409135, 0.5571109353025671, 0.5571109353025671, 0.5571109353025671, 1.3971332393348945, 3.9806464611637344, 2.8202775138618366, 0.7624436412130756, 1.1682169719523359, 0.6076255623768083, 0.6829564148915779, 0.6110741391877629, 0.5571372122780447, 0.5571365877795416, 0.5571310153313599, 0.5571190057447613, 0.5571186214379901, 0.5571174204793302, 0.5571172283259447, 0.5571136734883115, 6.580903986594154, 2.911180390972311, 1.7136526081928074, 1.7104681188625317, 1.2850260931423072, 1.2837064811796794, 1.2837064811796794, 1.2837064811796794, 0.9372041736009162, 0.7838592455112058, 1.2884688827559367, 1.2884984978098835, 0.5137605701373266, 0.5137605701373266, 0.5137604117680542, 0.5137589468522841, 0.5137569672363785, 0.5137532455584761, 0.5137529288199313, 0.513752136973569, 0.5137516222734336, 0.5135329934928266, 0.5135161667576296, 0.5134768515857457, 0.5134753074853394, 0.5128531537985487, 0.5127019507356841, 0.5125109969854365, 0.5125004258365009, 0.5124469762070516, 5.676861230057808, 0.6941498625511948, 1.2833597316576664, 1.2884660321090327, 2.5949451436349578, 0.9381267537975305, 3.0942986630965916, 0.93536629819422, 2.3031350308171525, 0.5137565317208793, 7.703042338353131, 3.2344229215773717, 2.3727243398018256, 1.9021834741740582, 1.9020777423407391, 1.9018990866400278, 2.3723463484977096, 1.041814703434485, 0.5705341814374543, 0.5705341425654568, 0.5705341425654568, 0.5705341036934592, 0.5705341036934592, 0.5705341036934592, 0.5705340648214616, 0.5705340648214616, 0.5705340648214616, 0.5705340648214616, 0.5705339870774666, 0.5705339870774666, 0.5705339870774666, 0.570533948205469, 0.5705339093334715, 0.5705339093334715, 0.5705339093334715, 0.570533870461474, 0.5705337538454813, 0.5705327820455427, 0.5705326654295501, 0.5705326654295501, 1.51308612938409, 2.8452909029655946, 1.0177982617040202, 0.5705354642133732, 0.5705346090294273, 0.5705346090294273, 3.58569028648644, 2.158663976869879, 1.4938678342544969, 1.493824270968449, 1.493461777805616, 1.1232914964105694, 1.1224175353023484, 1.1224175353023484, 1.1201093367765524, 1.1201093367765524, 1.1201093367765524, 1.1186412977457227, 1.1186412977457227, 1.1182711555109248, 1.1182711555109248, 1.1182711555109248, 1.1182711555109248, 1.1149528423959325, 1.0932339946104257, 4.567363440668118, 1.477938678311697, 1.4938672514680615, 0.4478674053897534, 0.4478673689656012, 0.4478673689656012, 0.4478673689656012, 0.44786733254144895, 0.44786733254144895, 0.44786733254144895, 0.44786733254144895, 5.187104655354188, 1.123290767927525, 4.711408432765805, 0.8246213887216183, 0.6329270491807271, 1.4244730066278235, 1.1516325272905696, 0.8661745846167597, 0.8661745846167597, 0.8661745846167597, 0.8661745846167597, 0.8661745846167597, 0.8661745846167597, 0.8661745846167597, 0.8661745846167597, 0.8660777415119021, 0.8647501706322221, 0.8647501706322221, 0.8647501706322221, 0.8647501706322221, 0.8647501706322221, 0.8647501706322221, 0.8637560639089821, 0.8637560639089821, 0.8620037257476004, 0.8620037257476004, 0.8620037257476004, 0.8597088957381062, 0.8597088957381062, 0.8597088957381062, 0.8597088957381062, 0.8597088957381062, 0.8670959258387702, 1.1304413595701206, 5.708436312291222, 1.1515876747634572, 1.8271572904321691, 0.8661750107690362, 0.8637551583353943, 0.8637456231782054, 1.5282568627362956, 1.0590217913779, 1.0586933069980393, 0.7970617762101748, 0.7970617762101748, 0.7958016705790703, 0.7958016705790703, 0.7958016705790703, 0.7958016705790703, 0.7958016705790703, 0.7954877035451899, 0.7954877035451899, 0.7944687428016419, 0.7944687428016419, 0.7944687428016419, 0.7944687428016419, 0.7941981708614221, 0.7941981708614221, 0.7941981708614221, 0.7941981708614221, 0.7941981708614221, 0.7941981708614221, 0.7399113795249345, 0.7944683504409397, 0.31773430478026293, 0.3177342851622278, 0.3171891980567314, 0.317185333303815, 0.3170414546343275, 0.3170241907634318, 0.795485192436696, 1.0178028086437663, 0.5082805327467617, 0.3177335200588586, 0.49168706851324834, 0.2692677581143414, 0.2692677581143414, 0.2692677220823956, 0.26926770406642264, 0.26926768605044976, 0.2692676680344768, 0.2692676500185039, 0.31557772641528686, 0.49551675990796745, 0.2911421398859791, 0.26926743382682894, 0.26926565024551025, 0.269265055718404, 0.046854901937731674, 0.046854879417765526, 0.046854879417765526, 0.04685486140179261, 0.046854879417765526, 0.04685487040977907, 0.04685487040977907, 0.04685487040977907, 0.04685494247367073, 0.046854946977663965, 0.04685491094571813, 0.04685488392175876, 0.046854901937731674, 0.046854892929745214, 0.04685488392175876, 0.046854879417765526, 0.26924095034664225, 0.04817961192225856, 0.04798548080610123, 0.04794498089898586, 0.04788455532582529, 0.04785278415758777, 0.047832110828666556, 0.047787188000200286, 0.047614297716111025, 0.0475840939375172, 0.04756269096169288, 0.04754361655036797, 0.0475156557604022, 0.047510111344737274, 0.04741020376693109, 0.04733104608593102, 0.05011030717179106, 0.047867548247392505, 0.04831314631351262, 0.0480452082603113, 0.047850338489264406, 0.04770679622505546, 0.047616590248664596], \"Total\": [8.0, 13.0, 12.0, 7.0, 12.0, 10.0, 7.0, 6.0, 4.0, 19.0, 3.0, 5.0, 7.0, 3.0, 2.0, 4.0, 2.0, 3.0, 3.0, 3.0, 3.0, 10.0, 4.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 2.0, 5.324886140605881, 4.916251020673946, 3.9255444630663594, 3.925323724390768, 3.924761249578502, 2.525640481133021, 2.5256382487167355, 2.5256315100382847, 2.5256315474633126, 2.525344698925982, 2.5237773981513842, 2.0290740491389414, 2.0286592141583006, 2.0286592141583006, 2.0285147980409217, 3.034697783647373, 3.0711269969330353, 1.6199400036402467, 1.6204428869199574, 2.9701741863912328, 3.6860279698640404, 1.1253340065265498, 1.1253340078296536, 1.125334008681186, 1.125333983941306, 1.1253339939284475, 1.1253340029454235, 1.1253339291668771, 1.1253339374907543, 1.1253339423549078, 10.187927750380915, 12.884145291810269, 10.95844843955414, 19.83997192810359, 13.425894040184, 2.712202445026832, 2.82112895624134, 7.16016870688823, 2.939279917415502, 3.692412998209928, 3.692115311157251, 3.691798093034482, 6.285631267775951, 2.3912422987566204, 2.3914315196670417, 4.186722170984551, 1.930287889078052, 1.9269647117640962, 1.550483861646516, 1.5504799279383092, 10.95844843955414, 3.4276455610007717, 12.262622221524552, 1.0902830080766843, 1.0902829602020758, 1.0902829720747114, 1.0902829798181548, 1.0902829847226636, 1.0902829524489845, 1.0902829591076137, 1.090282903188608, 1.090282880672444, 1.0902822768979683, 1.0902822774092635, 1.0902822281469944, 1.0902821137516308, 1.0902821148343307, 1.0902821340994404, 1.090282137941016, 2.7378075773360577, 19.83997192810359, 13.425894040184, 2.9701741863912328, 12.884145291810269, 1.5603195220584152, 10.187927750380915, 7.203460250687908, 1.9314782344545136, 1.7508497418369184, 2.7878062227090727, 1.5615351748321555, 1.5615350643398067, 1.4602167026970712, 1.7809769245516607, 1.5615361103845513, 7.597512750233303, 3.4534003527114012, 2.254397542543645, 2.254000042916207, 1.8294473648622285, 1.8295099080807529, 1.8295099080807529, 1.8295099080807529, 1.4784157844850223, 1.4197913834559381, 2.572039242014503, 2.6367264868347284, 1.054475745870198, 1.0544757615929907, 1.0544756827440849, 1.0544743414473317, 1.0544727925698647, 1.054469513909296, 1.0544694841245352, 1.0544693221602521, 1.0544691727462652, 1.0544754630045763, 1.054478214777857, 1.0544846164652166, 1.054484918760331, 1.054044088048158, 1.053972036616277, 1.0538809904438358, 1.053875979478581, 1.053850480921098, 12.884145291810269, 1.4470366546089029, 2.82112895624134, 2.8745386461229767, 7.203460250687908, 2.172524573773741, 10.187927750380915, 2.939279917415502, 19.83997192810359, 1.7881726464176266, 8.234740937952514, 3.765930198111416, 2.904506576600538, 2.4335691071674797, 2.4334571951008064, 2.4335971866616912, 3.27427608072059, 1.572651839882257, 1.101370490478444, 1.101370448370345, 1.101370479537725, 1.101370428656141, 1.1013704474175208, 1.101370455054689, 1.1013703971360609, 1.1013704260807708, 1.1013704325097737, 1.1013704459710167, 1.1013703264291097, 1.1013703281622023, 1.1013703350586814, 1.1013703312833825, 1.1013702825133054, 1.1013703049399506, 1.1013703143582154, 1.1013702994956283, 1.1013702013824536, 1.1013693304095127, 1.1013692615217108, 1.1013692681744636, 2.959688319911312, 7.203460250687908, 10.187927750380915, 1.7881726464176266, 2.172524573773741, 1.4713026056780882, 4.360559486408422, 2.7237281572407275, 2.0460778663512165, 2.046088746052032, 2.046140323476118, 1.675473828592038, 1.6757475709209235, 1.6757475709209235, 1.676172042685196, 1.676172042685196, 1.676172042685196, 1.6764615414333508, 1.6764615414333508, 1.6766559216273231, 1.6766559216273231, 1.6766559216273231, 1.6766559216273231, 1.6781959561737883, 1.6668580059565135, 7.16016870688823, 2.7878062227090727, 2.911779729003945, 1.0000486019324695, 1.0000485572531954, 1.0000485844639297, 1.000048588211977, 1.0000485849595138, 1.0000485887644475, 1.0000485927788634, 1.00004860256851, 12.262622221524552, 2.8745386461229767, 19.83997192810359, 3.6860279698640404, 2.3427942146386935, 2.0105684864780193, 1.721693102592394, 1.43621038375816, 1.43621038375816, 1.43621038375816, 1.43621038375816, 1.43621038375816, 1.43621038375816, 1.43621038375816, 1.43621038375816, 1.4361147332745043, 1.4368948949716287, 1.4368948949716287, 1.4368948949716287, 1.4368948949716287, 1.4368948949716287, 1.4368948949716287, 1.4376291925292937, 1.4376291925292937, 1.4387523188437374, 1.4387523188437374, 1.4387523188437374, 1.4362745853662195, 1.4362745853662195, 1.4362745853662195, 1.4362745853662195, 1.4362745853662195, 1.69906646258354, 2.232511542568196, 13.425894040184, 2.712202445026832, 19.83997192810359, 2.179617816881266, 2.6367264868347284, 2.7378075773360577, 2.120752775582674, 1.6347747583058927, 1.634767982342578, 1.371889246016662, 1.371889246016662, 1.3728461898320592, 1.3728461898320592, 1.3728461898320592, 1.3728461898320592, 1.3728461898320592, 1.3729729725860558, 1.3729729725860558, 1.3735231548690212, 1.3735231548690212, 1.3735231548690212, 1.3735231548690212, 1.3738066732952112, 1.3738066732952112, 1.3738066732952112, 1.3738066732952112, 1.3738066732952112, 1.3738066732952112, 1.4088479480203386, 2.179617816881266, 0.892561299467708, 0.8925613596010893, 0.8926084845674112, 0.8926088280319514, 0.8926209460835174, 0.8926223941373901, 2.572039242014503, 3.4276455610007717, 10.187927750380915, 1.7881726464176266, 1.074948502851522, 0.8525288635694019, 0.8525288690825235, 0.8525288567026016, 0.8525288605052328, 0.8525288504589814, 0.8525288670501292, 0.8525288729164964, 1.7508497418369184, 2.911779729003945, 1.7809769245516607, 2.172524573773741, 2.177822640590326, 4.360559486408422, 0.8925613596010893, 0.892561299467708, 0.8926084845674112, 0.8926088280319514, 0.8926209460835174, 0.8926223941373901, 0.8926352572423575, 0.8926441762470316, 0.9152962509651823, 0.9152971929026346, 0.9152972587797449, 0.9152972325762575, 0.9152977621967014, 0.915297768100612, 1.0000467865889917, 1.0000477823328613, 13.425894040184, 1.0532710810296273, 1.0534479328343667, 1.053484811472894, 1.053539922052028, 1.053568845907454, 1.0535876859068785, 1.0536285893112054, 1.0537861186493356, 1.0538135629442753, 1.0538331314548224, 1.053850480921098, 1.053875979478581, 1.0538809904438358, 1.053972036616277, 1.054044088048158, 2.959688319911312, 1.4197913834559381, 2.939279917415502, 7.203460250687908, 3.6860279698640404, 3.034697783647373, 1.4470366546089029], \"Category\": [\"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\"], \"logprob\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, -3.3407, -3.4298, -3.685, -3.6852, -3.6855, -4.2155, -4.2155, -4.2155, -4.2155, -4.2159, -4.2176, -4.5009, -4.5041, -4.5041, -4.5069, -4.1736, -4.1751, -4.8196, -4.8208, -4.3326, -4.2208, -5.4202, -5.4202, -5.4202, -5.4202, -5.4202, -5.4202, -5.4202, -5.4202, -5.4202, -3.3613, -3.1682, -3.4402, -2.9893, -3.5878, -4.818, -4.8123, -4.2209, -4.818, -3.4862, -3.4865, -3.4866, -2.9651, -4.0175, -4.0177, -3.4868, -4.3019, -4.3143, -4.6191, -4.6191, -2.8139, -3.9778, -2.7861, -5.2213, -5.2213, -5.2213, -5.2213, -5.2213, -5.2213, -5.2213, -5.2213, -5.2213, -5.2213, -5.2213, -5.2213, -5.2213, -5.2213, -5.2213, -5.2213, -4.3019, -3.2549, -3.5995, -4.9075, -4.4808, -5.1345, -5.0176, -5.1288, -5.2212, -5.2212, -5.2213, -5.2213, -5.2213, -5.2213, -5.2213, -5.2213, -2.5588, -3.3744, -3.9043, -3.9062, -4.1922, -4.1932, -4.1932, -4.1932, -4.5078, -4.6865, -4.1895, -4.1895, -5.1089, -5.1089, -5.1089, -5.1089, -5.1089, -5.1089, -5.1089, -5.1089, -5.1089, -5.1094, -5.1094, -5.1095, -5.1095, -5.1107, -5.111, -5.1114, -5.1114, -5.1115, -2.7065, -4.808, -4.1935, -4.1895, -3.4894, -4.5068, -3.3134, -4.5097, -3.6087, -5.1089, -2.383, -3.2507, -3.5605, -3.7816, -3.7816, -3.7817, -3.5607, -4.3836, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.9858, -4.0104, -3.3789, -4.4069, -4.9858, -4.9858, -4.9858, -3.0826, -3.59, -3.9582, -3.9582, -3.9584, -4.2433, -4.244, -4.244, -4.2461, -4.2461, -4.2461, -4.2474, -4.2474, -4.2477, -4.2477, -4.2477, -4.2477, -4.2507, -4.2704, -2.8406, -3.9689, -3.9582, -5.1628, -5.1628, -5.1628, -5.1628, -5.1628, -5.1628, -5.1628, -5.1628, -2.7134, -4.2433, -2.8095, -4.5524, -4.8169, -3.6927, -3.9053, -4.1902, -4.1902, -4.1902, -4.1902, -4.1902, -4.1902, -4.1902, -4.1902, -4.1903, -4.1918, -4.1918, -4.1918, -4.1918, -4.1918, -4.1918, -4.193, -4.193, -4.195, -4.195, -4.195, -4.1977, -4.1977, -4.1977, -4.1977, -4.1977, -4.1891, -3.9239, -2.3046, -3.9054, -3.4437, -4.1902, -4.193, -4.193, -3.3166, -3.6834, -3.6837, -3.9676, -3.9676, -3.9692, -3.9692, -3.9692, -3.9692, -3.9692, -3.9695, -3.9695, -3.9708, -3.9708, -3.9708, -3.9708, -3.9712, -3.9712, -3.9712, -3.9712, -3.9712, -3.9712, -4.042, -3.9708, -4.8873, -4.8873, -4.889, -4.889, -4.8895, -4.8895, -3.9695, -3.7231, -4.4175, -4.8873, -3.6723, -4.2745, -4.2745, -4.2745, -4.2745, -4.2745, -4.2745, -4.2745, -4.1158, -3.6646, -4.1964, -4.2745, -4.2745, -4.2745, -6.0231, -6.0231, -6.0231, -6.0231, -6.0231, -6.0231, -6.0231, -6.0231, -6.0231, -6.0231, -6.0231, -6.0231, -6.0231, -6.0231, -6.0231, -6.0231, -4.2746, -5.9952, -5.9993, -6.0001, -6.0014, -6.002, -6.0025, -6.0034, -6.007, -6.0077, -6.0081, -6.0085, -6.0091, -6.0092, -6.0113, -6.013, -5.9559, -6.0017, -5.9925, -5.998, -6.0021, -6.0051, -6.007], \"loglift\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, 1.3905, 1.3812, 1.3511, 1.3509, 1.3507, 1.2616, 1.2616, 1.2615, 1.2615, 1.2613, 1.2602, 1.1951, 1.1921, 1.1921, 1.1893, 1.1198, 1.1064, 1.1016, 1.1, 0.9824, 0.8783, 0.8653, 0.8653, 0.8653, 0.8653, 0.8653, 0.8653, 0.8653, 0.8653, 0.8653, 0.7211, 0.6794, 0.5693, 0.4266, 0.2186, 0.5879, 0.5541, 0.2142, 0.5074, 1.6111, 1.6109, 1.6109, 1.6002, 1.5143, 1.514, 1.4848, 1.444, 1.4334, 1.3459, 1.3459, 1.1956, 1.1939, 1.111, 1.0959, 1.0959, 1.0959, 1.0959, 1.0959, 1.0959, 1.0959, 1.0959, 1.0959, 1.0959, 1.0959, 1.0959, 1.0959, 1.0959, 1.0959, 1.0959, 1.0945, 0.161, 0.2069, 0.4074, -0.6332, 0.8242, -0.9352, -0.6998, 0.524, 0.6222, 0.1571, 0.7366, 0.7366, 0.8037, 0.6051, 0.7366, 1.817, 1.7898, 1.6864, 1.6847, 1.6074, 1.6063, 1.6063, 1.6063, 1.5048, 1.3666, 1.2694, 1.2446, 1.2416, 1.2416, 1.2416, 1.2416, 1.2416, 1.2416, 1.2416, 1.2416, 1.2416, 1.2412, 1.2411, 1.241, 1.241, 1.2402, 1.24, 1.2397, 1.2397, 1.2396, 1.141, 1.2261, 1.173, 1.1582, 0.9396, 1.1209, 0.769, 0.8157, -0.1928, 0.7134, 1.9123, 1.8269, 1.7768, 1.7326, 1.7326, 1.7325, 1.6568, 1.5672, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3213, 1.3081, 1.0501, -0.3246, 0.8366, 0.6419, 1.0317, 1.8484, 1.8115, 1.7295, 1.7295, 1.7292, 1.6442, 1.6433, 1.6433, 1.641, 1.641, 1.641, 1.6395, 1.6395, 1.639, 1.639, 1.639, 1.639, 1.6351, 1.6222, 1.5944, 1.4094, 1.3766, 1.2407, 1.2407, 1.2407, 1.2407, 1.2407, 1.2407, 1.2407, 1.2407, 1.1837, 1.1044, 0.6063, 0.5467, 0.7353, 2.0125, 1.9549, 1.8514, 1.8514, 1.8514, 1.8514, 1.8514, 1.8514, 1.8514, 1.8514, 1.8513, 1.8493, 1.8493, 1.8493, 1.8493, 1.8493, 1.8493, 1.8476, 1.8476, 1.8448, 1.8448, 1.8448, 1.8439, 1.8439, 1.8439, 1.8439, 1.8439, 1.6844, 1.6766, 1.5018, 1.5005, -0.0279, 1.4343, 1.2411, 1.2034, 2.3352, 2.2287, 2.2284, 2.1198, 2.1198, 2.1175, 2.1175, 2.1175, 2.1175, 2.1175, 2.117, 2.117, 2.1154, 2.1154, 2.1154, 2.1154, 2.1148, 2.1148, 2.1148, 2.1148, 2.1148, 2.1148, 2.0188, 1.6536, 1.6299, 1.6299, 1.6282, 1.6282, 1.6277, 1.6276, 1.4893, 1.4486, -0.3351, 0.9351, 2.659, 2.2887, 2.2887, 2.2887, 2.2887, 2.2887, 2.2887, 2.2887, 1.7277, 1.6702, 1.6301, 1.3532, 1.3508, 0.6565, 0.4941, 0.4941, 0.4941, 0.4941, 0.4941, 0.4941, 0.494, 0.494, 0.469, 0.469, 0.469, 0.469, 0.469, 0.469, 0.3804, 0.3804, -0.4682, 0.3564, 0.3522, 0.3514, 0.35, 0.3494, 0.3489, 0.3479, 0.3442, 0.3435, 0.343, 0.3426, 0.342, 0.3419, 0.3397, 0.3379, -0.6374, 0.0513, -0.6671, -1.569, -0.9031, -0.7116, 0.0271]}, \"token.table\": {\"Topic\": [3, 5, 7, 6, 6, 4, 4, 6, 2, 3, 3, 4, 1, 7, 4, 1, 5, 4, 5, 7, 3, 2, 2, 2, 1, 3, 4, 1, 6, 3, 6, 4, 5, 6, 7, 3, 1, 1, 2, 4, 6, 6, 1, 2, 6, 6, 7, 1, 6, 1, 2, 3, 4, 7, 2, 4, 1, 6, 2, 4, 4, 6, 3, 4, 3, 4, 6, 3, 6, 3, 1, 7, 3, 7, 2, 7, 1, 2, 3, 4, 1, 3, 3, 7, 3, 7, 4, 2, 3, 2, 3, 6, 3, 5, 7, 3, 1, 1, 3, 2, 2, 7, 1, 2, 1, 2, 5, 2, 1, 2, 1, 7, 3, 4, 1, 2, 3, 5, 6, 1, 2, 7, 6, 7, 5, 1, 5, 1, 2, 1, 2, 2, 4, 3, 5, 5, 4, 2, 3, 1, 2, 3, 3, 3, 4, 3, 3, 3, 5, 6, 4, 2, 4, 4, 3, 2, 4, 1, 3, 5, 1, 3, 1, 5, 1, 2, 5, 7, 3, 5, 1, 7, 6, 2, 1, 2, 3, 6, 1, 7, 4, 6, 6, 2, 6, 7, 6, 1, 2, 5, 2, 6, 1, 4, 4, 6, 6, 6, 4, 4, 4, 7, 2, 3, 4, 3, 7, 1, 5, 3, 3, 4, 2, 4, 7, 3, 5, 2, 3, 7, 2, 3, 6, 7, 4, 4, 2, 7, 5, 5, 2, 5, 2, 1, 3, 3, 4, 2, 1, 5, 2, 6, 6, 1, 1, 4, 2, 4, 3, 1, 1, 5, 2, 3, 7, 5, 5, 1, 6, 3, 6, 3, 4, 4, 4, 6, 3, 1, 3, 1, 1, 3, 4, 3, 2, 5, 2, 4, 1, 2, 2], \"Freq\": [0.9494231997924485, 0.5964252934075008, 0.9430613615255096, 0.6962768207978558, 0.6962768207978558, 0.9079606893857819, 0.9079598391730277, 0.6962456971589602, 0.8363191601147999, 0.948344461693076, 0.9483385360032416, 0.9714938284371967, 0.6173068124454306, 0.7279044566011614, 0.6796698355190663, 0.4929715084976308, 0.5965974700294003, 0.9079597109315454, 0.9173134806365416, 0.7284137199101164, 0.9492638115577656, 0.9171931404917297, 0.917193099051772, 0.9171938046007446, 0.3402193830110935, 0.3402193830110935, 0.3402193830110935, 0.4479260155804786, 0.4479260155804786, 0.5465944707831891, 0.6959451268839986, 0.9079597586791341, 0.7342876691578868, 0.695046664323472, 0.7279044566011614, 0.9489012133163036, 0.7919710924415947, 0.8886251153411215, 0.795465051479094, 0.9079606948702622, 0.6963231953758225, 0.497371766605043, 0.29793174205218, 0.223448806539135, 0.44689761307827, 0.4587960294024678, 0.4587960294024678, 0.49293641486005046, 0.6962456971589602, 0.49077693938426836, 0.09815538787685367, 0.29446616363056105, 0.09815538787685367, 0.09815538787685367, 0.9171930934502457, 0.9079599200566599, 0.6171152393409811, 0.6962768207978558, 0.8125423360787951, 0.907959732694267, 0.90796063807967, 0.6962768207978558, 0.6763997046665263, 0.9079598269204437, 0.9491548690761501, 0.907959834684777, 0.6959451268839986, 0.5465944707831891, 0.6959451268839986, 0.5466131571789209, 0.8136281046633088, 0.728054708400864, 0.9488782546260929, 0.7284137199101164, 0.9171938036899292, 0.728054708400864, 0.1388221722892832, 0.1388221722892832, 0.41646651686784963, 0.41646651686784963, 0.3544680216718362, 0.3544680216718362, 0.3887965563141129, 0.3887965563141129, 0.9483303006131333, 0.7097998058663203, 0.821828694971298, 0.9171931594333013, 0.9483441816528639, 0.6848298599467909, 0.9492305813141092, 0.6962768207978558, 0.9489168352673583, 0.5999311257626637, 0.7279044566011614, 0.9483385501434721, 0.8886250591252735, 0.7642681752230809, 0.5465944707831891, 0.5711512393695428, 0.583490901963638, 0.291745450981819, 0.8886250597976899, 0.9171930760276448, 0.42684073306635695, 0.42684073306635695, 0.42684073306635695, 0.6449615902668997, 0.6512267327262238, 0.3256133663631119, 0.8886251115001204, 0.7284137199101164, 0.8871549769981529, 0.8218759730093176, 0.3528230798595236, 0.20161318849115636, 0.10080659424557818, 0.25201648561394546, 0.10080659424557818, 0.36501517729117006, 0.5475227659367551, 0.6117076005899181, 0.695046664323472, 0.7289218155937456, 0.5964944469558271, 0.8886250636545442, 0.5958779702222351, 0.5614895882223392, 0.5614895882223392, 0.45917421435610456, 0.45917421435610456, 0.640395436566126, 0.640395436566126, 0.9483441548657409, 0.5967485899146733, 0.5964252934075008, 0.7966159334298006, 0.836385338717011, 0.9488737429250488, 0.6733611816989103, 0.3366805908494552, 0.6910674977133006, 0.9489591695150866, 0.5592301179661656, 0.5592301179661656, 0.9483305724763849, 0.9483386069156933, 0.34343257150913603, 0.34343257150913603, 0.695046664323472, 0.9079598098551652, 0.6403950528904101, 0.6403950528904101, 0.6885851166984854, 0.8873114294232294, 0.5177381666340233, 0.5177381666340233, 0.5425894801535568, 0.2712947400767784, 0.2712947400767784, 0.7918783433114697, 0.9487269188632944, 0.8886251219141095, 0.5965974700294003, 0.9389872136178832, 0.3587049888382278, 0.3587049888382278, 0.7279044566011614, 0.8687090095547657, 0.4887250344107065, 0.7642251993896946, 0.7283464568981686, 0.6962768207978558, 0.6449599539449983, 0.46568863235451596, 0.077614772059086, 0.46568863235451596, 0.6959451268839986, 0.8886250786612315, 0.7283464568981686, 0.6358686485082858, 0.6959451268839986, 0.6955896591391898, 0.917193092529537, 0.5885584949275237, 0.7279044566011614, 0.6962456971589602, 0.9856712724943285, 0.9171937842515584, 0.5968460879155221, 0.917193708366287, 0.6959451268839986, 0.88862507077484, 0.8218381775596557, 0.6108220414815624, 0.6962768207978558, 0.6962456971589602, 0.6955896591391898, 0.9079597172275569, 0.9079597164420573, 0.9079598169693067, 0.728054708400864, 0.5189508629270749, 0.9483388045376074, 0.9079596907479552, 0.9487917755488547, 0.7279044566011614, 0.49293641486005046, 0.4887373541003632, 0.7043288272153639, 0.4602939879584283, 0.4602939879584283, 0.9171930719017605, 0.9079598129674877, 0.7284137199101164, 0.347881911884798, 0.347881911884798, 0.8124768278777027, 0.9491010495963627, 0.7289218155937456, 0.9171936669246858, 0.9483443273165474, 0.580823608164705, 0.728054708400864, 0.9079598531731032, 0.9079597348173786, 0.9171930522553514, 0.7284137199101164, 0.5964944469558271, 0.5967485899146733, 0.9171937874832624, 0.5964252934075008, 0.91719366735481, 0.6590442088754617, 0.32952210443773083, 0.9491809271472638, 0.9079597184200416, 0.9171930825417636, 0.7643777058597344, 0.5964252934075008, 0.36525576460454545, 0.36525576460454545, 0.6962768207978558, 0.7918790432541915, 0.8886250608266917, 0.9079598155405607, 0.6403954818797392, 0.6403954818797392, 0.9483398132072496, 0.7918811443453638, 0.27932302741358017, 0.6983075685339504, 0.5180574388194613, 0.9489344559260328, 0.6117050651285404, 0.5965974700294003, 0.4887399528852273, 0.3687040404500878, 0.3687040404500878, 0.3792581464148961, 0.3792581464148961, 0.9213541628850896, 0.1316220232692985, 0.9079597295173673, 0.9079596817285273, 0.6962456971589602, 0.9491378965190232, 0.7924629174763826, 0.9483363297464293, 0.7918811560795277, 0.3378734150053899, 0.3378734150053899, 0.6757468300107798, 0.9483412061897694, 0.4892917592672972, 0.40774313272274765, 0.6408943718660735, 0.6408943718660735, 0.23885033665007674, 0.7165510099502301, 0.8126121538608151], \"Term\": [\"academy\", \"accept\", \"achieve\", \"achievement\", \"acknowledge\", \"act\", \"activity\", \"actually\", \"advancement\", \"advantage\", \"alone\", \"also\", \"always\", \"appreciation\", \"area\", \"arebite\", \"arrive\", \"associate\", \"atmosphere\", \"attention\", \"available\", \"average\", \"base\", \"basis\", \"bechtle\", \"bechtle\", \"bechtle\", \"brand\", \"brand\", \"brange\", \"breakfast\", \"brilliant\", \"building\", \"canteen\", \"capitalise\", \"centrally\", \"chance\", \"cheerful\", \"climate\", \"close\", \"coherence\", \"cohesion\", \"colleague\", \"colleague\", \"colleague\", \"commitment\", \"commitment\", \"communication\", \"compact\", \"company\", \"company\", \"company\", \"company\", \"company\", \"compensate\", \"concept\", \"constant\", \"constructively\", \"contract\", \"contribution\", \"corporation\", \"criticism\", \"customer\", \"decentralise\", \"decentralize\", \"decision\", \"decisionmake\", \"default\", \"design\", \"develop\", \"development\", \"door\", \"due\", \"dynamic\", \"economic\", \"effort\", \"employee\", \"employee\", \"employee\", \"employee\", \"employer\", \"employer\", \"employment\", \"employment\", \"encouragement\", \"enormous\", \"environment\", \"especially\", \"evolution\", \"excellent\", \"explain\", \"express\", \"eye\", \"familiar\", \"feedback\", \"feel\", \"financially\", \"find\", \"fixedterm\", \"flat\", \"flexible\", \"flexible\", \"flexibly\", \"food\", \"free\", \"free\", \"free\", \"friendly\", \"future\", \"future\", \"futureoriente\", \"futureproof\", \"give\", \"goal\", \"good\", \"good\", \"good\", \"good\", \"good\", \"great\", \"great\", \"group\", \"hard\", \"hardware\", \"harmonious\", \"help\", \"helpful\", \"hierarchy\", \"hierarchy\", \"high\", \"high\", \"house\", \"house\", \"htle\", \"humanly\", \"improvement\", \"individual\", \"innovative\", \"interact\", \"interesting\", \"interesting\", \"internal\", \"know\", \"large\", \"large\", \"leader\", \"leave\", \"level\", \"level\", \"likely\", \"little\", \"live\", \"live\", \"make\", \"management\", \"manager\", \"manager\", \"many\", \"many\", \"many\", \"market\", \"measure\", \"mood\", \"much\", \"new\", \"nice\", \"nice\", \"objective\", \"offer\", \"office\", \"open\", \"openende\", \"openly\", \"openness\", \"opportunity\", \"opportunity\", \"opportunity\", \"option\", \"organize\", \"outset\", \"part\", \"path\", \"pay\", \"performance\", \"perhaps\", \"person\", \"personality\", \"perspective\", \"pioneering\", \"place\", \"portfolio\", \"position\", \"positioning\", \"positive\", \"possibility\", \"praise\", \"promise\", \"promote\", \"promotion\", \"quick\", \"realise\", \"really\", \"reasonable\", \"recognition\", \"regard\", \"regionally\", \"regular\", \"relax\", \"reputation\", \"resource\", \"responsibility\", \"responsibility\", \"restaurant\", \"retrospect\", \"right\", \"safe\", \"safe\", \"salary\", \"scene\", \"sell\", \"service\", \"share\", \"short\", \"show\", \"simply\", \"skill\", \"slightly\", \"slogan\", \"small\", \"specialist\", \"stable\", \"stateoftheart\", \"strength\", \"strong\", \"strong\", \"structure\", \"structured\", \"subsidize\", \"successful\", \"suggestion\", \"super\", \"super\", \"superior\", \"supervisor\", \"sustainable\", \"switch\", \"system\", \"system\", \"take\", \"task\", \"team\", \"team\", \"teamwork\", \"theoretically\", \"thing\", \"though\", \"time\", \"top\", \"top\", \"train\", \"train\", \"training\", \"training\", \"transfer\", \"trust\", \"unusual\", \"use\", \"usually\", \"various\", \"vary\", \"well\", \"well\", \"well\", \"wellknown\", \"work\", \"work\", \"working\", \"working\", \"working_hour\", \"working_hour\", \"young\"]}, \"R\": 30, \"lambda.step\": 0.01, \"plot.opts\": {\"xlab\": \"PC1\", \"ylab\": \"PC2\"}, \"topic.order\": [7, 1, 5, 8, 4, 3, 6, 2]};\n",
       "\n",
       "function LDAvis_load_lib(url, callback){\n",
       "  var s = document.createElement('script');\n",
       "  s.src = url;\n",
       "  s.async = true;\n",
       "  s.onreadystatechange = s.onload = callback;\n",
       "  s.onerror = function(){console.warn(\"failed to load library \" + url);};\n",
       "  document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "}\n",
       "\n",
       "if(typeof(LDAvis) !== \"undefined\"){\n",
       "   // already loaded: just create the visualization\n",
       "   !function(LDAvis){\n",
       "       new LDAvis(\"#\" + \"ldavis_el3072424679739468324926357982\", ldavis_el3072424679739468324926357982_data);\n",
       "   }(LDAvis);\n",
       "}else if(typeof define === \"function\" && define.amd){\n",
       "   // require.js is available: use it to load d3/LDAvis\n",
       "   require.config({paths: {d3: \"https://d3js.org/d3.v5\"}});\n",
       "   require([\"d3\"], function(d3){\n",
       "      window.d3 = d3;\n",
       "      LDAvis_load_lib(\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v3.0.0.js\", function(){\n",
       "        new LDAvis(\"#\" + \"ldavis_el3072424679739468324926357982\", ldavis_el3072424679739468324926357982_data);\n",
       "      });\n",
       "    });\n",
       "}else{\n",
       "    // require.js not available: dynamically load d3 & LDAvis\n",
       "    LDAvis_load_lib(\"https://d3js.org/d3.v5.js\", function(){\n",
       "         LDAvis_load_lib(\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v3.0.0.js\", function(){\n",
       "                 new LDAvis(\"#\" + \"ldavis_el3072424679739468324926357982\", ldavis_el3072424679739468324926357982_data);\n",
       "            })\n",
       "         });\n",
       "}\n",
       "</script>"
      ],
      "text/plain": [
       "PreparedData(topic_coordinates=              x         y  topics  cluster       Freq\n",
       "topic                                                \n",
       "6     -0.091493  0.103996       1        1  22.424872\n",
       "0     -0.111473 -0.013937       2        1  17.079729\n",
       "4      0.045251  0.125821       3        1  14.076797\n",
       "7      0.117331  0.040953       4        1  13.820692\n",
       "3     -0.097935 -0.024013       5        1  12.950376\n",
       "2     -0.022625 -0.157017       6        1   9.469734\n",
       "5      0.137989 -0.058257       7        1   6.975068\n",
       "1      0.022954 -0.017547       8        1   3.202732, topic_info=            Term       Freq      Total Category  logprob  loglift\n",
       "153         also   8.000000   8.000000  Default  30.0000  30.0000\n",
       "39     colleague  13.000000  13.000000  Default  29.0000  29.0000\n",
       "45          work  12.000000  12.000000  Default  28.0000  28.0000\n",
       "68      training   7.000000   7.000000  Default  27.0000  27.0000\n",
       "99   opportunity  12.000000  12.000000  Default  26.0000  26.0000\n",
       "..           ...        ...        ...      ...      ...      ...\n",
       "17       bechtle   0.048313   2.939280   Topic8  -5.9925  -0.6671\n",
       "0       employee   0.048045   7.203460   Topic8  -5.9980  -1.5690\n",
       "60          many   0.047850   3.686028   Topic8  -6.0021  -0.9031\n",
       "26        strong   0.047707   3.034698   Topic8  -6.0051  -0.7116\n",
       "57      internal   0.047617   1.447037   Topic8  -6.0070   0.0271\n",
       "\n",
       "[348 rows x 6 columns], token_table=      Topic      Freq          Term\n",
       "term                               \n",
       "48        3  0.949423       academy\n",
       "276       5  0.596425        accept\n",
       "254       7  0.943061       achieve\n",
       "282       6  0.696277   achievement\n",
       "283       6  0.696277   acknowledge\n",
       "...     ...       ...           ...\n",
       "46        2  0.640894       working\n",
       "46        4  0.640894       working\n",
       "47        1  0.238850  working_hour\n",
       "47        2  0.716551  working_hour\n",
       "131       2  0.812612         young\n",
       "\n",
       "[281 rows x 3 columns], R=30, lambda_step=0.01, plot_opts={'xlab': 'PC1', 'ylab': 'PC2'}, topic_order=[7, 1, 5, 8, 4, 3, 6, 2])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# In this cell, we provide the company's dataframe for which we would be extracting topics. The motiavtion is to run it on a yearly basis and \n",
    "# analyse the topic topics that have been spoken about. We run the cell below from the year 2010-2022. By doing so, as analysts, we \n",
    "# analyse words belonging to various unsupervised clusters and generate the most spoken about topics. From our analysis we were able to identify, 3 \n",
    "# main topics - 1. Management, 2. Remunerations and 3. Work Environement. The bubbles are unsupervised clusters and an approximation of key topics.\n",
    "# Through, running multiple interations and analyzing the various bubbles and perplexity, we were able to bucket the reviews to the aforementioned \n",
    "# 3 topics\n",
    "# By analyzing all clusters from 2010-2022, we were able to identify these topics\n",
    "topic_modelling(input_df_time,2016,'Gut am Arbeitgeber finde ich_plain_text')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### End of Notebook\n",
    "\n",
    "##### Next notebook is 04_word_frequency.ipynb"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}