04_embeddings_deep_dive.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Personality and its Transformations ###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An analysis of prof. Jordan Peterson's collection of lectures from University of Toronto personality course."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|    More about prof. Peterson at https://www.jordanbpeterson.com/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "        <script type=\"text/javascript\">\n",
       "        window.PlotlyConfig = {MathJaxConfig: 'local'};\n",
       "        if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}\n",
       "        if (typeof require !== 'undefined') {\n",
       "        require.undef(\"plotly\");\n",
       "        requirejs.config({\n",
       "            paths: {\n",
       "                'plotly': ['https://cdn.plot.ly/plotly-latest.min']\n",
       "            }\n",
       "        });\n",
       "        require(['plotly'], function(Plotly) {\n",
       "            window._Plotly = Plotly;\n",
       "        });\n",
       "        }\n",
       "        </script>\n",
       "        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import plotly.express as px\n",
    "from plotly.offline import plot, iplot, init_notebook_mode\n",
    "import plotly.graph_objs as go\n",
    "import plotly.io as pio\n",
    "init_notebook_mode(connected=True)\n",
    "\n",
    "from sentence_transformers import SentenceTransformer\n",
    "\n",
    "from constants import *\n",
    "\n",
    "import umap\n",
    "import hdbscan\n",
    "from collections import Counter\n",
    "from tqdm import tqdm\n",
    "\n",
    "import nltk\n",
    "from gensim.utils import simple_preprocess\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"tomcolor8\">  \n",
    "<h4 style=\"background:#135e96; color:white ;font-size:15px;line-height:1em; text-align:left; padding: 20px\">\n",
    "      Load the data</h4> \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(f'{OUTPUT_FOLDER}\\{FINAL_CAPTIONS_FILE_NAME}', 'r', encoding='utf-8') as f:\n",
    "    data = f.readlines()\n",
    "data = [sent.replace('\\n', '') for sent in data]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Well, after all that.',\n",
       " 'So, welcome to Psychology 230.',\n",
       " 'Nice to see you all here.',\n",
       " 'So, what I’m going to do today—how I’m going to start—is I’m going to give you an overview of the content of the course',\n",
       " 'and then I’ll give you an overview of the class requirements right at the end.']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"tomcolor8\">  \n",
    "<h4 style=\"background:#135e96; color:white ;font-size:15px;line-height:1em; text-align:left; padding: 20px\">\n",
    "      Check the dataset</h4> \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6144"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NO_OF_SENTS = len(data)\n",
    "NO_OF_SENTS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 6144 sentences in total"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "LONGEST_SENTENCE = 315"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Maximal number of words in a single sentence is 315 (information obtained from notebook 02). This is because oral lectures typically are chains of thoughts from the speaker and don't follow a syntactic structure typical for written information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"tomcolor8\">  \n",
    "<h4 style=\"background:#135e96; color:white ;font-size:15px;line-height:1em; text-align:left; padding: 20px\">\n",
    "      Model implementation</h4> \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A paraphrase identification model was chosen because generally models like those are good at similarity and retrieval tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = 'paraphrase-distilroberta-base-v1'\n",
    "model = SentenceTransformer(model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The chooses model is a STS model that was trained on NLI Data and fine-tuned on a STS benchmark dataset. This means that the model is suitable for semantic similarity cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "128"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.max_seq_length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default model has length 128, but it must be increased because our longest sentence has 315 words and the number of tokens will be even greater. A max value of 512 will be set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.max_seq_length = SENT_LENGTH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SentenceTransformer(\n",
       "  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel \n",
       "  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})\n",
       ")"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This  model uses a mean pooling layer. Mean-pooling approach has the highest performences on NLI and STSb datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally the model is a Roberta-based model which ensures better results over models like Bert and XLNet on  GLUE benchmatk since Roberta was trained on a larger dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"tomcolor8\">  \n",
    "<h4 style=\"background:#135e96; color:white ;font-size:15px;line-height:1em; text-align:left; padding: 20px\">\n",
    "      Make basic embeddings</h4> \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c8ebb8cfcc254a988efd32a371abe663",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/48 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 4min 24s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "embeddings = model.encode(data, batch_size=128, convert_to_numpy=True, show_progress_bar=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(6144, 768) <class 'numpy.ndarray'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[ 0.01237977,  0.2777765 , -0.01405313, ..., -0.02678866,\n",
       "         0.04404815,  0.07773356],\n",
       "       [ 0.04129101,  0.16136684,  0.02991311, ...,  0.58379793,\n",
       "         0.14942269,  0.06346932],\n",
       "       [-0.07501772,  0.4233094 , -0.0145095 , ...,  0.33547857,\n",
       "         0.41762486, -0.00107458],\n",
       "       ...,\n",
       "       [ 0.00656719,  0.14959459,  0.08193731, ...,  0.16275299,\n",
       "         0.10600478,  0.02682212],\n",
       "       [ 0.40873346, -0.28501287,  1.0132743 , ...,  0.09641485,\n",
       "         0.09259868, -0.08356816],\n",
       "       [ 0.40873346, -0.28501287,  1.0132743 , ...,  0.09641485,\n",
       "         0.09259868, -0.08356816]], dtype=float32)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(embeddings.shape, type(embeddings))\n",
    "embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving the model ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(f'{MODELS_FOLDER}\\{model_name}.npy', 'wb') as f:\n",
    "    np.save(f, embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the model ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with open(f'{MODELS_FOLDER}\\{SAMPLE_EMBEDDINGS_MODEL_NAME}.npy', 'rb') as f:\n",
    "#     embeddings = np.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"tomcolor8\">  \n",
    "<h4 style=\"background:#135e96; color:white ;font-size:15px;line-height:1em; text-align:left; padding: 20px\">\n",
    "      Dimensionality reduction</h4> \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***UMAP*** for a dimensionality reduction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "neighbors = 20\n",
    "components = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`n_neighbors` regulates whether UMAP should focus more on global or lobal scructures of the data. Low values of n_neighbors will force UMAP to concentrate on very local structure (it will find patterns in local small sections not seeing the big picture), while large values will push UMAP to look at larger neighborhoods (focusing on the big picture) losing fine distinct features of local, smaller neighborhoods. You can think of this like literally focusing on your neighbors when you live for example on a country side. If n_neighbors is small someone can describe you  based on your membership to only your small community (e.g. people from your street). If n_neighbors is large someone can describe you based on the judgement from people from the whole city or state."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`n_components` the dimensionality of the reduced dimension space we will be embedding the data into."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be trying to reduce 768 dimmentions to 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the sample model ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "UMAP(angular_rp_forest=True, learning_rate=0.4, metric='cosine', min_dist=0.0, n_components=10, n_epochs=4000, n_neighbors=20, verbose=True)\n",
      "Thu Apr 27 21:45:45 2023 Construct fuzzy simplicial set\n",
      "Thu Apr 27 21:45:45 2023 Finding Nearest Neighbors\n",
      "Thu Apr 27 21:45:45 2023 Building RP forest with 9 trees\n",
      "Thu Apr 27 21:45:46 2023 NN descent for 13 iterations\n",
      "\t 1  /  13\n",
      "\t 2  /  13\n",
      "\t 3  /  13\n",
      "\t 4  /  13\n",
      "\t 5  /  13\n",
      "\t 6  /  13\n",
      "\tStopping threshold met -- exiting after 6 iterations\n",
      "Thu Apr 27 21:46:02 2023 Finished Nearest Neighbor Search\n",
      "Thu Apr 27 21:46:07 2023 Construct embedding\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e2a56343dcd04fc2973cfd38c01c9af8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epochs completed:   0%|            0/4000 [00:00]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thu Apr 27 21:47:10 2023 Finished embedding\n",
      "Wall time: 1min 25s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "umap_embeddings = umap.UMAP(\n",
    "    n_neighbors = neighbors,\n",
    "    n_components = components,\n",
    "    n_epochs = 4000,\n",
    "    min_dist = 0.0,\n",
    "    learning_rate = 0.4,\n",
    "    verbose = True,\n",
    "    metric = 'cosine').fit_transform(embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(6144, 10) <class 'numpy.ndarray'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[ 2.7284317 ,  0.73799795, -0.89301753, ...,  1.031198  ,\n",
       "         8.286052  ,  7.6264105 ],\n",
       "       [ 2.9306781 ,  0.7101402 , -0.696889  , ...,  1.165728  ,\n",
       "         8.730799  ,  7.6870384 ],\n",
       "       [ 2.890705  ,  0.7187636 , -0.70411944, ...,  1.1766963 ,\n",
       "         8.642087  ,  7.6341186 ],\n",
       "       ...,\n",
       "       [ 2.463766  ,  0.7197509 , -1.0211723 , ...,  1.0415711 ,\n",
       "         7.615919  ,  7.906527  ],\n",
       "       [ 5.05041   ,  0.21676305,  0.40999976, ...,  0.87336725,\n",
       "        11.482289  ,  8.363158  ],\n",
       "       [ 5.0504656 ,  0.21699952,  0.4099645 , ...,  0.87367743,\n",
       "        11.4824295 ,  8.363     ]], dtype=float32)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(umap_embeddings.shape, type(umap_embeddings))\n",
    "umap_embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"tomcolor8\">  \n",
    "<h4 style=\"background:#135e96; color:white ;font-size:15px;line-height:1em; text-align:left; padding: 20px\">\n",
    "      Clustering</h4> \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***HDBSCAN*** is a hierarchical clustering algorithm. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'euclidean': hdbscan.dist_metrics.EuclideanDistance,\n",
       " 'l2': hdbscan.dist_metrics.EuclideanDistance,\n",
       " 'minkowski': hdbscan.dist_metrics.MinkowskiDistance,\n",
       " 'p': hdbscan.dist_metrics.MinkowskiDistance,\n",
       " 'manhattan': hdbscan.dist_metrics.ManhattanDistance,\n",
       " 'cityblock': hdbscan.dist_metrics.ManhattanDistance,\n",
       " 'l1': hdbscan.dist_metrics.ManhattanDistance,\n",
       " 'chebyshev': hdbscan.dist_metrics.ChebyshevDistance,\n",
       " 'infinity': hdbscan.dist_metrics.ChebyshevDistance,\n",
       " 'seuclidean': hdbscan.dist_metrics.SEuclideanDistance,\n",
       " 'mahalanobis': hdbscan.dist_metrics.MahalanobisDistance,\n",
       " 'wminkowski': hdbscan.dist_metrics.WMinkowskiDistance,\n",
       " 'hamming': hdbscan.dist_metrics.HammingDistance,\n",
       " 'canberra': hdbscan.dist_metrics.CanberraDistance,\n",
       " 'braycurtis': hdbscan.dist_metrics.BrayCurtisDistance,\n",
       " 'matching': hdbscan.dist_metrics.MatchingDistance,\n",
       " 'jaccard': hdbscan.dist_metrics.JaccardDistance,\n",
       " 'dice': hdbscan.dist_metrics.DiceDistance,\n",
       " 'kulsinski': hdbscan.dist_metrics.KulsinskiDistance,\n",
       " 'rogerstanimoto': hdbscan.dist_metrics.RogersTanimotoDistance,\n",
       " 'russellrao': hdbscan.dist_metrics.RussellRaoDistance,\n",
       " 'sokalmichener': hdbscan.dist_metrics.SokalMichenerDistance,\n",
       " 'sokalsneath': hdbscan.dist_metrics.SokalSneathDistance,\n",
       " 'haversine': hdbscan.dist_metrics.HaversineDistance,\n",
       " 'cosine': hdbscan.dist_metrics.ArccosDistance,\n",
       " 'arccos': hdbscan.dist_metrics.ArccosDistance,\n",
       " 'pyfunc': hdbscan.dist_metrics.PyFuncDistance}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hdbscan.dist_metrics.METRIC_MAPPING"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_cluser_size = 80\n",
    "min_samples = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`min_cluster_size` - the minimum size a final cluster can be.   \n",
    "`min_samples` - the higher this is, the more points are going to be treeted as outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 589 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "clusters = hdbscan.HDBSCAN(\n",
    "    min_cluster_size = min_cluser_size,\n",
    "    metric = 'euclidean',\n",
    "    min_samples = min_samples,\n",
    "    cluster_selection_epsilon = 0.1,\n",
    "    cluster_selection_method = 'leaf',\n",
    "    leaf_size = 35,\n",
    "    algorithm = 'best').fit(umap_embeddings)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cluster_statistics(clusters:hdbscan.hdbscan_.HDBSCAN, \n",
    "compact : bool = False) -> pd.DataFrame:\n",
    "    \"\"\"Print basic statistic about a given structure of clusters\n",
    "        - number of unique clusters\n",
    "        - number of all points\n",
    "        - number of points treeted as outliers (noise)\n",
    "        This also plots distribution of the clusters\n",
    "\n",
    "    Args:\n",
    "        clusters (hdbscan.hdbscan_.HDBSCAN): HDBSCAN clusters \n",
    "        compact (bool): if True only two values (number of clusters and \n",
    "        percentage of noise) will be outputed\n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: a dataframe with basic statistics\n",
    "        Additionaly a plot of cluster distribution in shown.\n",
    "    \"\"\"\n",
    "\n",
    "    # empty dataframe\n",
    "    summary = pd.DataFrame(columns = ['metric', 'value'])\n",
    "    summary.at[0, 'metric'] = 'Number of unique clusters'\n",
    "\n",
    "    # get number of all clusters in first row\n",
    "    summary.at[0, 'value'] = len(set(clusters.labels_))\n",
    "\n",
    "    # count number of points in each cluster (sentences in each group)\n",
    "    cnt = Counter()\n",
    "    cnt.update(clusters.labels_)\n",
    "\n",
    "    # total number of all sentences\n",
    "    sum_of_all_elements = sum([entry[1] for entry in cnt.most_common()])\n",
    "    # number of outlier sentences\n",
    "    sum_of_empty_elements = [entry[1] for entry in cnt.most_common() if entry[0] == -1][0]\n",
    "\n",
    "    # get total number of all sentences and number of outliers into the dataframe\n",
    "    summary.at[1, 'metric'] = 'Number of all sentences'\n",
    "    summary.at[1, 'value'] = sum_of_all_elements\n",
    "    \n",
    "    summary.at[2, 'metric'] = 'Number of noise'\n",
    "\n",
    "    # calculate how many percent of total sentences are the outlier sentences\n",
    "    percentage = '{:.4}'.format(str((sum_of_empty_elements/sum_of_all_elements)*100))\n",
    "    summary.at[2, 'value'] = f'{sum_of_empty_elements} ({percentage}%)'\n",
    "\n",
    "    # get points for the distribution showing number of sentences in each cluster\n",
    "    x, y= zip(*cnt.most_common())\n",
    "\n",
    "    # output plot and dataframe as summay\n",
    "    if compact != True:\n",
    "        plt.bar(x, y);\n",
    "        return summary\n",
    "\n",
    "    # output only number of clusters and percentage of noise\n",
    "    else:\n",
    "        return len(set(clusters.labels_)), sum_of_empty_elements/sum_of_all_elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>metric</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Number of unique clusters</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Number of all sentences</td>\n",
       "      <td>6144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Number of noise</td>\n",
       "      <td>2269 (36.9%)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      metric         value\n",
       "0  Number of unique clusters            22\n",
       "1    Number of all sentences          6144\n",
       "2            Number of noise  2269 (36.9%)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAANNUlEQVR4nO3dX6icd53H8fdnW90LFUxpNpQ0u3ElLGQvNpZQC1uWitCm7UUqLMVe2CCVeJGCgjdZbypKoTfqUnALkYZG0EpBuw1r2BqC4O5F3Z5K6R+7ktBNaUKaxI1UQdil+t2L8wvOJufknJw/c077fb/gMM/85pk5v/N0eM+TZ56ZpqqQJPXwJ2s9AUnS9Bh9SWrE6EtSI0Zfkhox+pLUyLVrPYEruf7662vr1q1rPQ1Jeld54YUXflVVG+e6bV1Hf+vWrczMzKz1NCTpXSXJG/Pd5uEdSWrE6EtSI0Zfkhox+pLUiNGXpEaMviQ1YvQlqRGjL0mNGH1JamRdfyJ3ubbu/9Gi1z35yN2rOBNJWh/c05ekRoy+JDVi9CWpEaMvSY0YfUlqxOhLUiNGX5IaMfqS1IjRl6RGjL4kNWL0JakRoy9JjRh9SWrE6EtSI0Zfkhox+pLUiNGXpEaMviQ1YvQlqRGjL0mNLBj9JFuS/CTJL5K8muQLY/y6JEeTHB+XG8Z4kjya5ESSl5LcNPFYe8b6x5PsWb0/S5I0l8Xs6b8DfKmqtgO3APuSbAf2A8eqahtwbFwHuBPYNn72Ao/B7IsE8BDwceBm4KGLLxSSpOlYMPpVdaaqfj6Wfwu8BmwGdgOHxmqHgHvG8m7gOzXrOeDDSW4A7gCOVtWFqvo1cBTYtZJ/jCTpyq7qmH6SrcDHgJ8Bm6rqzLjpLWDTWN4MvDlxt1NjbL7xS3/H3iQzSWbOnz9/NdOTJC1g0dFP8kHgB8AXq+o3k7dVVQG1EhOqqgNVtbOqdm7cuHElHlKSNCwq+knex2zwv1tVPxzDZ8dhG8bluTF+Gtgycfcbx9h845KkKVnM2TsBHgdeq6pvTNx0GLh4Bs4e4JmJ8fvHWTy3AG+Pw0DPArcn2TDewL19jEmSpuTaRazzt8BngJeTvDjGvgw8AjyV5AHgDeDecdsR4C7gBPA74LMAVXUhydeA58d6X62qCyvxR0iSFmfB6FfVvwOZ5+ZPzrF+AfvmeayDwMGrmaAkaeX4iVxJasToS1IjRl+SGjH6ktSI0ZekRoy+JDVi9CWpEaMvSY0YfUlqxOhLUiNGX5IaMfqS1IjRl6RGjL4kNWL0JakRoy9JjRh9SWrE6EtSI0Zfkhox+pLUiNGXpEaMviQ1YvQlqRGjL0mNGH1JasToS1IjRl+SGjH6ktSI0ZekRoy+JDVi9CWpEaMvSY0YfUlqxOhLUiNGX5IaMfqS1IjRl6RGFox+koNJziV5ZWLsK0lOJ3lx/Nw1cds/JDmR5JdJ7pgY3zXGTiTZv/J/iiRpIYvZ038C2DXH+Derasf4OQKQZDvwaeCvx33+Kck1Sa4BvgXcCWwH7hvrSpKm6NqFVqiqnybZusjH2w18v6r+B/ivJCeAm8dtJ6rqdYAk3x/r/uLqpyxJWqrlHNN/MMlL4/DPhjG2GXhzYp1TY2y+8csk2ZtkJsnM+fPnlzE9SdKllhr9x4CPAjuAM8DXV2pCVXWgqnZW1c6NGzeu1MNKkljE4Z25VNXZi8tJvg38y7h6GtgyseqNY4wrjEuSpmRJe/pJbpi4+ing4pk9h4FPJ/nTJB8BtgH/ATwPbEvykSTvZ/bN3sNLn7YkaSkW3NNP8iRwG3B9klPAQ8BtSXYABZwEPg9QVa8meYrZN2jfAfZV1e/H4zwIPAtcAxysqldX+o+RJF3ZYs7euW+O4cevsP7DwMNzjB8BjlzV7CRJK8pP5EpSI0Zfkhox+pLUiNGXpEaMviQ1YvQlqRGjL0mNGH1JasToS1IjRl+SGjH6ktSI0ZekRoy+JDVi9CWpEaMvSY0YfUlqxOhLUiNGX5IaMfqS1IjRl6RGjL4kNWL0JakRoy9JjRh9SWrE6EtSI0Zfkhox+pLUiNGXpEaMviQ1YvQlqRGjL0mNGH1JasToS1IjRl+SGjH6ktSI0ZekRhaMfpKDSc4leWVi7LokR5McH5cbxniSPJrkRJKXktw0cZ89Y/3jSfaszp8jSbqSxezpPwHsumRsP3CsqrYBx8Z1gDuBbeNnL/AYzL5IAA8BHwduBh66+EIhSZqeBaNfVT8FLlwyvBs4NJYPAfdMjH+nZj0HfDjJDcAdwNGqulBVvwaOcvkLiSRplS31mP6mqjozlt8CNo3lzcCbE+udGmPzjUuSpmjZb+RWVQG1AnMBIMneJDNJZs6fP79SDytJYunRPzsO2zAuz43x08CWifVuHGPzjV+mqg5U1c6q2rlx48YlTk+SNJelRv8wcPEMnD3AMxPj94+zeG4B3h6HgZ4Fbk+yYbyBe/sYkyRN0bULrZDkSeA24Pokp5g9C+cR4KkkDwBvAPeO1Y8AdwEngN8BnwWoqgtJvgY8P9b7alVd+uawJGmVLRj9qrpvnps+Oce6Beyb53EOAgevanaSpBXlJ3IlqRGjL0mNGH1JasToS1IjRl+SGjH6ktSI0ZekRoy+JDVi9CWpEaMvSY0YfUlqxOhLUiNGX5IaMfqS1IjRl6RGjL4kNWL0JakRoy9JjRh9SWrE6EtSI0Zfkhox+pLUiNGXpEaMviQ1YvQlqRGjL0mNGH1JasToS1IjRl+SGjH6ktSI0ZekRoy+JDVi9CWpEaMvSY1cu9YTEGzd/6NFr3vykbtXcSaS3uvc05ekRoy+JDWyrOgnOZnk5SQvJpkZY9clOZrk+LjcMMaT5NEkJ5K8lOSmlfgDJEmLtxJ7+p+oqh1VtXNc3w8cq6ptwLFxHeBOYNv42Qs8tgK/W5J0FVbj8M5u4NBYPgTcMzH+nZr1HPDhJDeswu+XJM1judEv4MdJXkiyd4xtqqozY/ktYNNY3gy8OXHfU2Ps/0myN8lMkpnz588vc3qSpEnLPWXz1qo6neTPgKNJ/nPyxqqqJHU1D1hVB4ADADt37ryq+0qSrmxZe/pVdXpcngOeBm4Gzl48bDMuz43VTwNbJu5+4xiTJE3JkqOf5ANJPnRxGbgdeAU4DOwZq+0BnhnLh4H7x1k8twBvTxwGkiRNwXIO72wCnk5y8XG+V1X/muR54KkkDwBvAPeO9Y8AdwEngN8Bn13G75YkLcGSo19VrwN/M8f4fwOfnGO8gH1L/X2SpOXzE7mS1IjRl6RGjL4kNWL0JakRoy9JjRh9SWrE6EtSI0Zfkhox+pLUiNGXpEaMviQ1YvQlqRGjL0mNLPf/nCVJl9m6/0eLXvfkI3ev4kx0KaMvvYsYUy2X0dd7ilGUrszoSw0s9cXQF9H3Ht/IlaRGjL4kNWL0JakRoy9JjRh9SWrEs3e0Ll3NWSPgmSPvFZ4ttPrc05ekRtzT16K45633oo7/snBPX5IaMfqS1IiHdySm/8/8jocVtD64py9Jjbinr1XlG8CaBv/ltHhGX1oGY6OrsR6eL0Z/DuvhP4yk9evd3Aijv4LeDW8GerhF6s03ciWpEaMvSY0YfUlqxOhLUiNTj36SXUl+meREkv3T/v2S1NlUo5/kGuBbwJ3AduC+JNunOQdJ6mzae/o3Ayeq6vWq+l/g+8DuKc9BktpKVU3vlyV/D+yqqs+N658BPl5VD06ssxfYO67+FfDLqU3wyq4HfrXWk1iH3C5zc7vMze1yudXYJn9RVRvnumHdfTirqg4AB9Z6HpdKMlNVO9d6HuuN22Vubpe5uV0uN+1tMu3DO6eBLRPXbxxjkqQpmHb0nwe2JflIkvcDnwYOT3kOktTWVA/vVNU7SR4EngWuAQ5W1avTnMMyrLtDTuuE22Vubpe5uV0uN9VtMtU3ciVJa8tP5EpSI0Zfkhox+ovgV0fMLcnJJC8neTHJzFrPZ60kOZjkXJJXJsauS3I0yfFxuWEt5zht82yTryQ5PZ4vLya5ay3nuBaSbEnykyS/SPJqki+M8ak9X4z+AvzqiAV9oqp2ND/3+glg1yVj+4FjVbUNODaud/IEl28TgG+O58uOqjoy5TmtB+8AX6qq7cAtwL7Rk6k9X4z+wvzqCF1RVf0UuHDJ8G7g0Fg+BNwzzTmttXm2SXtVdaaqfj6Wfwu8Bmxmis8Xo7+wzcCbE9dPjTFBAT9O8sL4+gz90aaqOjOW3wI2reVk1pEHk7w0Dv+0OuR1qSRbgY8BP2OKzxejr+W4tapuYvbQ174kf7fWE1qPava8aM+NhseAjwI7gDPA19d0NmsoyQeBHwBfrKrfTN622s8Xo78wvzpiHlV1elyeA55m9lCYZp1NcgPAuDy3xvNZc1V1tqp+X1V/AL5N0+dLkvcxG/zvVtUPx/DUni9Gf2F+dcQcknwgyYcuLgO3A69c+V6tHAb2jOU9wDNrOJd14WLUhk/R8PmSJMDjwGtV9Y2Jm6b2fPETuYswTi37R/741REPr+2M1l6Sv2R27x5mv87je123S5IngduY/Yrcs8BDwD8DTwF/DrwB3FtVbd7YnGeb3MbsoZ0CTgKfnziO3UKSW4F/A14G/jCGv8zscf2pPF+MviQ14uEdSWrE6EtSI0Zfkhox+pLUiNGXpEaMviQ1YvQlqZH/AybIceSbmvI1AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cluster_statistics(clusters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"tomcolor8\">  \n",
    "<h4 style=\"background:#135e96; color:white ;font-size:15px;line-height:1em; text-align:left; padding: 20px\">\n",
    "      Cluster analysis</h4> \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_data = pd.DataFrame(data, columns=['text'])\n",
    "final_data['cluster_ids'] = clusters.labels_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>cluster_ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Well, after all that.</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>So, welcome to Psychology 230.</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Nice to see you all here.</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>So, what I’m going to do today—how I’m going t...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>and then I’ll give you an overview of the clas...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6139</th>\n",
       "      <td>And if you're on the low end of the IQ distrib...</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6140</th>\n",
       "      <td>It has nothing to do with willingness to work,...</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6141</th>\n",
       "      <td>And it's a good thing to know, even though it'...</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6142</th>\n",
       "      <td></td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6143</th>\n",
       "      <td></td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6144 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   text  cluster_ids\n",
       "0                                 Well, after all that.           18\n",
       "1                        So, welcome to Psychology 230.           11\n",
       "2                             Nice to see you all here.           11\n",
       "3     So, what I’m going to do today—how I’m going t...            3\n",
       "4     and then I’ll give you an overview of the clas...            3\n",
       "...                                                 ...          ...\n",
       "6139  And if you're on the low end of the IQ distrib...           13\n",
       "6140  It has nothing to do with willingness to work,...           10\n",
       "6141  And it's a good thing to know, even though it'...           17\n",
       "6142                                                              -1\n",
       "6143                                                              -1\n",
       "\n",
       "[6144 rows x 2 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cluster_ids</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-1</td>\n",
       "      <td>So, there’s a website—I don’t really like Blac...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>That’s a reasonable way of thinking about it, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>So when we’re first born—we’re very primitive ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>And so in the mirror test what you essentially...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>So, what I’m going to do today—how I’m going t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4</td>\n",
       "      <td>So the first issue is that there’s a lot of re...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>5</td>\n",
       "      <td>But I think we might as well jump right into t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>6</td>\n",
       "      <td>Now, part of the reason for that is that Nietz...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>7</td>\n",
       "      <td>It brings in elements of cultural history, ele...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>8</td>\n",
       "      <td>Most of the brain is structured with the olfac...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>9</td>\n",
       "      <td>Now, dolphins seem to be able to manage that, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>10</td>\n",
       "      <td>But they like things cut and dried and laid ou...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>11</td>\n",
       "      <td>So, welcome to Psychology 230. Nice to see you...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>12</td>\n",
       "      <td>And useful, like there’s a difference between ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>13</td>\n",
       "      <td>You know and that requires a fair bit of intel...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>14</td>\n",
       "      <td>You don’t really have to learn what anger mean...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>15</td>\n",
       "      <td>So, personality is a somewhat peculiar field o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>16</td>\n",
       "      <td>You think in terms of the presence of positive...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>17</td>\n",
       "      <td>You know, you’re really really really really c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>18</td>\n",
       "      <td>Well, after all that. And, so, some people wil...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>19</td>\n",
       "      <td>Now, the problem is—or partly is—is that you d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>20</td>\n",
       "      <td>You can even use your body as a representation...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    cluster_ids                                               text\n",
       "0            -1  So, there’s a website—I don’t really like Blac...\n",
       "1             0  That’s a reasonable way of thinking about it, ...\n",
       "2             1  So when we’re first born—we’re very primitive ...\n",
       "3             2  And so in the mirror test what you essentially...\n",
       "4             3  So, what I’m going to do today—how I’m going t...\n",
       "5             4  So the first issue is that there’s a lot of re...\n",
       "6             5  But I think we might as well jump right into t...\n",
       "7             6  Now, part of the reason for that is that Nietz...\n",
       "8             7  It brings in elements of cultural history, ele...\n",
       "9             8  Most of the brain is structured with the olfac...\n",
       "10            9  Now, dolphins seem to be able to manage that, ...\n",
       "11           10  But they like things cut and dried and laid ou...\n",
       "12           11  So, welcome to Psychology 230. Nice to see you...\n",
       "13           12  And useful, like there’s a difference between ...\n",
       "14           13  You know and that requires a fair bit of intel...\n",
       "15           14  You don’t really have to learn what anger mean...\n",
       "16           15  So, personality is a somewhat peculiar field o...\n",
       "17           16  You think in terms of the presence of positive...\n",
       "18           17  You know, you’re really really really really c...\n",
       "19           18  Well, after all that. And, so, some people wil...\n",
       "20           19  Now, the problem is—or partly is—is that you d...\n",
       "21           20  You can even use your body as a representation..."
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emb = final_data.groupby(['cluster_ids'], as_index=False).agg({'text': ' '.join})\n",
    "emb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "emb.to_csv(f'{OUTPUT_FOLDER}\\{CLUSTERED_TOPICS_DATAFRAME_NAME}.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"tomcolor8\">  \n",
    "<h4 style=\"background:#135e96; color:white ;font-size:15px;line-height:1em; text-align:left; padding: 20px\">\n",
    "      Visualisation</h4> \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data needs to be reduced to 3 dimmentions for visualisation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 1min 44s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "umap_3d = umap.UMAP(\n",
    "    n_neighbors = neighbors,\n",
    "    n_components = 3,\n",
    "    min_dist = 0.5,\n",
    "    n_epochs = 8000,\n",
    "    learning_rate = 0.4,\n",
    "    metric = 'cosine').fit_transform(embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show all reduced representations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-49.21519  ,  11.144993 ,  11.598254 ],\n",
       "       [-48.308266 ,  11.1203375,  11.974965 ],\n",
       "       [-48.738823 ,  10.80503  ,  11.731585 ],\n",
       "       ...,\n",
       "       [-50.281292 ,  11.445325 ,  12.993492 ],\n",
       "       [-47.204277 ,   9.426386 ,   9.995483 ],\n",
       "       [-47.227512 ,   9.453953 ,  10.045561 ]], dtype=float32)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "umap_3d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[18, 11, 11, 3, 3]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster_list = list(clusters.labels_)\n",
    "cluster_list[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6144, 3)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "umap_3d.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>cluster</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-49.215191</td>\n",
       "      <td>11.144993</td>\n",
       "      <td>11.598254</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-48.308266</td>\n",
       "      <td>11.120337</td>\n",
       "      <td>11.974965</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-48.738823</td>\n",
       "      <td>10.805030</td>\n",
       "      <td>11.731585</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-47.686726</td>\n",
       "      <td>12.897839</td>\n",
       "      <td>13.580678</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-47.546715</td>\n",
       "      <td>12.775138</td>\n",
       "      <td>13.494473</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6139</th>\n",
       "      <td>-47.382633</td>\n",
       "      <td>11.306064</td>\n",
       "      <td>16.297216</td>\n",
       "      <td>13.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6140</th>\n",
       "      <td>-50.624229</td>\n",
       "      <td>10.122428</td>\n",
       "      <td>14.016111</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6141</th>\n",
       "      <td>-50.281292</td>\n",
       "      <td>11.445325</td>\n",
       "      <td>12.993492</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6142</th>\n",
       "      <td>-47.204277</td>\n",
       "      <td>9.426386</td>\n",
       "      <td>9.995483</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6143</th>\n",
       "      <td>-47.227512</td>\n",
       "      <td>9.453953</td>\n",
       "      <td>10.045561</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6144 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              x          y          z  cluster\n",
       "0    -49.215191  11.144993  11.598254     18.0\n",
       "1    -48.308266  11.120337  11.974965     11.0\n",
       "2    -48.738823  10.805030  11.731585     11.0\n",
       "3    -47.686726  12.897839  13.580678      3.0\n",
       "4    -47.546715  12.775138  13.494473      3.0\n",
       "...         ...        ...        ...      ...\n",
       "6139 -47.382633  11.306064  16.297216     13.0\n",
       "6140 -50.624229  10.122428  14.016111     10.0\n",
       "6141 -50.281292  11.445325  12.993492     17.0\n",
       "6142 -47.204277   9.426386   9.995483     -1.0\n",
       "6143 -47.227512   9.453953  10.045561     -1.0\n",
       "\n",
       "[6144 rows x 4 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster_3d_df = pd.DataFrame(np.insert(umap_3d, 3, cluster_list, axis=1), columns=['x', 'y', 'z', 'cluster'])\n",
    "cluster_3d_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simple plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = px.scatter_3d(cluster_3d_df, x = 'x', y = 'y', z = 'z', color='cluster')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "pio.write_html(fig, file=f'{OUTPUT_FOLDER}\\{EMBEDDING_CHART_NAME}.html', auto_open=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Preview at:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://rawcdn.githack.com/MariuszAndziak/Personality_and_Its_Transformations/bd84d60093e7e328a82debb0154ceaa7ffea1f45/outputs/23_embeddings_chart.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "f5dc7221bd57f443bd21d92d893b3958d3081f94c6c945d5f95188cb4cde5b4f"
  },
  "kernelspec": {
   "display_name": "Python 3.7.5 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}