ItEM.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "<link rel=\"stylesheet\" href=\"https://cdn.jupyter.org/notebook/5.1.0/style/style.min.css\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ItEM - Italian EMotive lexicon\n",
    "\n",
    "ItEM is a a high-coverage emotion lexicon for Italian in which each target term is provided with an association score with the basic emotions defined the in Plutchik (1994)’s taxonomy: **JOY, SADNESS, ANGER, FEAR, TRUST, DISGUST, SURPRISE, ANTICIPATION**.\n",
    "\n",
    "![alt text](images/1181px-Plutchik-wheel.png \"Plutchik's wheel of emotions\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents of this repository\n",
    " \n",
    " - the list of seed words collected in Passaro et al., 2015. The seeds are provided both as lemmas and tokens;\n",
    " \n",
    " - the pre-compiled emotive lexicon described in Passaro and Lenci (2016) and referred as _SintParModel_;\n",
    " \n",
    " - the pre-compiled emotive lexicon built by exploiting count vectors extracted from FB-NEWS15 (Passaro et al., 2016); \n",
    " \n",
    " - a simplified implementation of ItEM that can be used to create new lexica from a list of seeds and a list of word embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### The list of seed words collected in Passaro et al., 2015"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Seed</th>\n",
       "      <th>Emotion</th>\n",
       "      <th>PoS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>colorato</td>\n",
       "      <td>gioia</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>esuberante</td>\n",
       "      <td>gioia</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>gaudio</td>\n",
       "      <td>gioia</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>esultanza</td>\n",
       "      <td>gioia</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>libero</td>\n",
       "      <td>gioia</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>550</th>\n",
       "      <td>anticipazione</td>\n",
       "      <td>attese</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>551</th>\n",
       "      <td>prospettiva</td>\n",
       "      <td>attese</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>552</th>\n",
       "      <td>sognare</td>\n",
       "      <td>attese</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>553</th>\n",
       "      <td>attesa</td>\n",
       "      <td>attese</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>554</th>\n",
       "      <td>supporre</td>\n",
       "      <td>attese</td>\n",
       "      <td>V</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>555 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              Seed Emotion PoS\n",
       "0         colorato   gioia   A\n",
       "1       esuberante   gioia   A\n",
       "2           gaudio   gioia   S\n",
       "3        esultanza   gioia   S\n",
       "4           libero   gioia   A\n",
       "..             ...     ...  ..\n",
       "550  anticipazione  attese   S\n",
       "551    prospettiva  attese   S\n",
       "552        sognare  attese   A\n",
       "553         attesa  attese   S\n",
       "554       supporre  attese   V\n",
       "\n",
       "[555 rows x 3 columns]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Markdown, display\n",
    "import pandas as pd\n",
    "\n",
    "display(Markdown(f'### The list of seed words collected in Passaro et al., 2015'))\n",
    "pd.read_csv(\"seeds/ItEM.elicitated.lemmas.txt\", sep = \"\\t\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### The pre-compiled emotive lexicon described in Passaro and Lenci (2016) referred as _SintParModel_"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>emotion</th>\n",
       "      <th>word</th>\n",
       "      <th>cosine</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>gioia</td>\n",
       "      <td>gioia-s</td>\n",
       "      <td>0.843964</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>gioia</td>\n",
       "      <td>giubilo-s</td>\n",
       "      <td>0.795281</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>gioia</td>\n",
       "      <td>letizia-s</td>\n",
       "      <td>0.779542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>gioia</td>\n",
       "      <td>esultanza-s</td>\n",
       "      <td>0.758201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>gioia</td>\n",
       "      <td>gaudio-s</td>\n",
       "      <td>0.754662</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239931</th>\n",
       "      <td>attese</td>\n",
       "      <td>folico-a</td>\n",
       "      <td>0.001367</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239932</th>\n",
       "      <td>attese</td>\n",
       "      <td>rilevante-a</td>\n",
       "      <td>0.000935</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239933</th>\n",
       "      <td>attese</td>\n",
       "      <td>necessario-a</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239934</th>\n",
       "      <td>attese</td>\n",
       "      <td>compensativi-a</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239935</th>\n",
       "      <td>attese</td>\n",
       "      <td>integrativo-a</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>239936 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       emotion            word    cosine\n",
       "0        gioia         gioia-s  0.843964\n",
       "1        gioia       giubilo-s  0.795281\n",
       "2        gioia       letizia-s  0.779542\n",
       "3        gioia     esultanza-s  0.758201\n",
       "4        gioia        gaudio-s  0.754662\n",
       "...        ...             ...       ...\n",
       "239931  attese        folico-a  0.001367\n",
       "239932  attese     rilevante-a  0.000935\n",
       "239933  attese    necessario-a  0.000000\n",
       "239934  attese  compensativi-a  0.000000\n",
       "239935  attese   integrativo-a  0.000000\n",
       "\n",
       "[239936 rows x 3 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display(Markdown(f'### The pre-compiled emotive lexicon described in Passaro and Lenci (2016) referred as _SintParModel_'))\n",
    "pd.read_csv(\"pre-compiled-lexica/ItEM.SintParModel.cos\", sep = \"\\t\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### The pre-compiled emotive lexicon built from FB-NEWS15 (Passaro et al., 2016)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>emotion</th>\n",
       "      <th>word</th>\n",
       "      <th>cosine</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>gioia</td>\n",
       "      <td>festoso-a</td>\n",
       "      <td>0.647874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>gioia</td>\n",
       "      <td>euforico-a</td>\n",
       "      <td>0.622582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>gioia</td>\n",
       "      <td>esilarante-a</td>\n",
       "      <td>0.622579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>gioia</td>\n",
       "      <td>gaio-a</td>\n",
       "      <td>0.617334</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>gioia</td>\n",
       "      <td>divertito-a</td>\n",
       "      <td>0.614806</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239941</th>\n",
       "      <td>attese</td>\n",
       "      <td>muitos-s</td>\n",
       "      <td>0.004284</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239942</th>\n",
       "      <td>attese</td>\n",
       "      <td>faço-s</td>\n",
       "      <td>0.003667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239943</th>\n",
       "      <td>attese</td>\n",
       "      <td>caminho-s</td>\n",
       "      <td>0.001646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239944</th>\n",
       "      <td>attese</td>\n",
       "      <td>weapons-s</td>\n",
       "      <td>0.001640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239945</th>\n",
       "      <td>attese</td>\n",
       "      <td>crê-s</td>\n",
       "      <td>0.000451</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>239946 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       emotion          word    cosine\n",
       "0        gioia     festoso-a  0.647874\n",
       "1        gioia    euforico-a  0.622582\n",
       "2        gioia  esilarante-a  0.622579\n",
       "3        gioia        gaio-a  0.617334\n",
       "4        gioia   divertito-a  0.614806\n",
       "...        ...           ...       ...\n",
       "239941  attese      muitos-s  0.004284\n",
       "239942  attese        faço-s  0.003667\n",
       "239943  attese     caminho-s  0.001646\n",
       "239944  attese     weapons-s  0.001640\n",
       "239945  attese         crê-s  0.000451\n",
       "\n",
       "[239946 rows x 3 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display(Markdown(f'### The pre-compiled emotive lexicon built from FB-NEWS15 (Passaro et al., 2016)'))\n",
    "pd.read_csv(\"pre-compiled-lexica/ItEM.FBNEWS15.cos\", sep = \"\\t\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Citation \n",
    "\n",
    "\n",
    "```\n",
    "@inproceedings{DBLP:conf/lrec/PassaroL16,\n",
    "  author    = {Lucia C. Passaro and\n",
    "               Alessandro Lenci},\n",
    "  editor    = {Nicoletta Calzolari and\n",
    "               Khalid Choukri and\n",
    "               Thierry Declerck and\n",
    "               Sara Goggi and\n",
    "               Marko Grobelnik and\n",
    "               Bente Maegaard and\n",
    "               Joseph Mariani and\n",
    "               H{\\'{e}}l{\\`{e}}ne Mazo and\n",
    "               Asunci{\\'{o}}n Moreno and\n",
    "               Jan Odijk and\n",
    "               Stelios Piperidis},\n",
    "  title     = {Evaluating Context Selection Strategies to Build Emotive Vector Space\n",
    "               Models},\n",
    "  booktitle = {Proceedings of the Tenth International Conference on Language Resources\n",
    "               and Evaluation {LREC} 2016, Portoro{\\v{z}}, Slovenia, May 23-28, 2016},\n",
    "  publisher = {European Language Resources Association {(ELRA)}},\n",
    "  year      = {2016},\n",
    "  url       = {http://www.lrec-conf.org/proceedings/lrec2016/summaries/637.html},\n",
    "  timestamp = {Mon, 19 Aug 2019 15:22:52 +0200},\n",
    "  biburl    = {https://dblp.org/rec/conf/lrec/PassaroL16.bib},\n",
    "  bibsource = {dblp computer science bibliography, https://dblp.org}\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create your own emotive lexicon:\n",
    "\n",
    "This section provides a simplified implementation of ItEM. Starting from a list of seed words and word embeddings for the desired vocabulary, you will be guided through the creation of a new emotive lexicon. You can build your vector space as you wish (both count and prediction-based embeddings) or you can use pre-trained vectors.\n",
    "The only constraint is that the seed words must be part of the vector space model of choice (i.e. you need to have the embeddings for such words in your space).\n",
    "\n",
    "\n",
    "### what you need"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. import code and libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "unable to import 'smart_open.gcs', disabling that module\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import Markdown, display\n",
    "import pandas as pd\n",
    "\n",
    "from utils import item_functions as item"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. a list of seed words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- A list of seed lemmas or tokens in a tab-separated file.\n",
    "- The PoS column is optional, but depending on its presence, the seed key (i.e. the unique key used to relate a word and its embedding) will be formatted in a different way. \n",
    "  - if populated, the seed key will be formatted as \"Seed-PoS\" (e.g. \"agitarsi-v\");\n",
    "  - otherwise, the seed key will be the \"Seed\" column as is (e.g. \"agitarsi\"). \n",
    "- It is clear that, depending on the seed keys (\"Seed-PoS\" vs \"Seed\"), the words in the vector space must be formatted in the same way to access the embeddings properly. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "#### A dafaframe containing seed lemmas"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Seed</th>\n",
       "      <th>Emotion</th>\n",
       "      <th>PoS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>colorato</td>\n",
       "      <td>gioia</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>esuberante</td>\n",
       "      <td>gioia</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>gaudio</td>\n",
       "      <td>gioia</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>esultanza</td>\n",
       "      <td>gioia</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>libero</td>\n",
       "      <td>gioia</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>550</th>\n",
       "      <td>anticipazione</td>\n",
       "      <td>attese</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>551</th>\n",
       "      <td>prospettiva</td>\n",
       "      <td>attese</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>552</th>\n",
       "      <td>sognare</td>\n",
       "      <td>attese</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>553</th>\n",
       "      <td>attesa</td>\n",
       "      <td>attese</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>554</th>\n",
       "      <td>supporre</td>\n",
       "      <td>attese</td>\n",
       "      <td>V</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>555 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              Seed Emotion PoS\n",
       "0         colorato   gioia   A\n",
       "1       esuberante   gioia   A\n",
       "2           gaudio   gioia   S\n",
       "3        esultanza   gioia   S\n",
       "4           libero   gioia   A\n",
       "..             ...     ...  ..\n",
       "550  anticipazione  attese   S\n",
       "551    prospettiva  attese   S\n",
       "552        sognare  attese   A\n",
       "553         attesa  attese   S\n",
       "554       supporre  attese   V\n",
       "\n",
       "[555 rows x 3 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display(Markdown(f'#### A dafaframe containing seed lemmas'))\n",
    "pd.read_csv(\"seeds/ItEM.elicitated.lemmas.txt\", sep = \"\\t\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "#### A dafaframe containing seed tokens"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Seed</th>\n",
       "      <th>Emotion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>colorato</td>\n",
       "      <td>gioia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>colorata</td>\n",
       "      <td>gioia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>colorate</td>\n",
       "      <td>gioia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>colorati</td>\n",
       "      <td>gioia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>esuberantissimo</td>\n",
       "      <td>gioia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8108</th>\n",
       "      <td>supponessimo</td>\n",
       "      <td>attese</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8109</th>\n",
       "      <td>supporrebbero</td>\n",
       "      <td>attese</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8110</th>\n",
       "      <td>supporrà</td>\n",
       "      <td>attese</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8111</th>\n",
       "      <td>supponevi</td>\n",
       "      <td>attese</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8112</th>\n",
       "      <td>supposi</td>\n",
       "      <td>attese</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8113 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Seed Emotion\n",
       "0            colorato   gioia\n",
       "1            colorata   gioia\n",
       "2            colorate   gioia\n",
       "3            colorati   gioia\n",
       "4     esuberantissimo   gioia\n",
       "...               ...     ...\n",
       "8108     supponessimo  attese\n",
       "8109    supporrebbero  attese\n",
       "8110         supporrà  attese\n",
       "8111        supponevi  attese\n",
       "8112          supposi  attese\n",
       "\n",
       "[8113 rows x 2 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display(Markdown(f'#### A dafaframe containing seed tokens'))\n",
    "\n",
    "pd.read_csv(\"seeds/ItEM.elicitated.tokens.txt\", sep = \"\\t\")    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. word vectors\n",
    "\n",
    "The file of word embeddings must be in the `word2vec C` format:\n",
    "\n",
    " - the first row contains the number of embeddings and their dimension;\n",
    " - each row is space-separated and contains the word in position 0, followed by its vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "29992 300\n",
       "\n",
       "abate-s -0.505393460602235 -3.0702947386964445 [...] 1.1265833671394883\n",
       "\n",
       "abbaglio-s -0.5159974598163771 -2.5505094487548488 [...] 0.24188720476360617\n",
       "\n",
       "abbandono-s -0.0903591412413367 -0.5987769584177935 [...] 0.022376339858185584"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(f'29992 300\\n\\nabate-s -0.505393460602235 -3.0702947386964445 [...] 1.1265833671394883\\n\\nabbaglio-s -0.5159974598163771 -2.5505094487548488 [...] 0.24188720476360617\\n\\nabbandono-s -0.0903591412413367 -0.5987769584177935 [...] 0.022376339858185584'))\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The file of the seeds  and vectors formatted as described above.\n",
    "seedsFile = \"seeds/ItEM.elicitated.lemmas.txt\"\n",
    "vectorsFile = \"vectors/COUNT.repubblica.itwac.TOP240.vec\"\n",
    "\n",
    "#other available options here:\n",
    "#seedsFile = \"seeds/ItEM.elicitated.tokens.txt\"\n",
    "#vectorsFile = \"vectors/PREDICT-wiki.it.vec\"\n",
    "\n",
    "# The cosine similarity threshold used to write the output.\n",
    "cosine_threshold = 0.2\n",
    "\n",
    "# The output file formatted as the pre-compiled lexica described above.\n",
    "outputFile = f\"output/ItEM.{cosine_threshold}.cos\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load word embeddings \n",
    "\n",
    "Word embeddings are loaded with the `load_vectors` function of `item`. The function takes as input file of embeddings in the `word2vec C format` and loads them in a Gensim `KeyedVectors` object. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorModel = item.load_vectors(vectorsFile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the centroid vectors\n",
    "\n",
    "First, seeds are loaded via the `item.load_seeds` function, that takes as input a seeds file such as the ones described above and returns a dictionary in which the key is the centroid name and the value is a set of seeds associated to that centroid.\n",
    "\n",
    "Then, the centroid vectors are computed with the `item.get_gentroid_vectors` function. The function takes as argument the dictionary `centroids` and the vector space model `vectorModel`. \n",
    "\n",
    "The output of the function are 3 aligned lists of length equal to the number of distinct centroid keys provided in the seeds file:\n",
    "\n",
    "1. indexes of the centroids\n",
    "2. names of the centroids (e.g. emotions)\n",
    "3. vectors for the centroids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "#### Emotive centroids and the number of seeds used to generate them: \n",
       "\n",
       "[('gioia', 61), ('rabbia', 77), ('sorpresa', 60), ('disgusto', 80), ('paura', 78), ('tristezza', 77), ('fiducia', 62), ('attese', 60)]\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "centroids = item.load_seeds(seedsFile)\n",
    "centroidIdxs, centroidNames, centroidVectors = item.get_centroid_vectors(centroids, vectorModel)\n",
    "display(Markdown(f'#### Emotive centroids and the number of seeds used to generate them: \\n\\n{[(k, len(centroids[k])) for k in centroids]}\\n'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute the similarity matrix\n",
    "\n",
    "Finally, the similarity between each target word in the vector space and each of the centroid is computed via the `item.get_similatity_matrix`.\n",
    "The function takes as argument the vector model and the centroid vectors computed above.\n",
    "The function returns the similarity matrix and 3 aligned lists of length equal to the number of distinct words belonging to the vector space model.\n",
    "\n",
    "1. the similarity matrix of shape (n_words,n_centroids)\n",
    "2. list of word embeddings in the model\n",
    "3. list of ids for word embeddings in the model\n",
    "4. list of words in the model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "#### The Similarity matrix"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "array([[ 0.03573099,  0.12702479,  0.00631439, ...,  0.03567612,\n",
       "        -0.02278734,  0.04787419],\n",
       "       [ 0.27003698,  0.17806558,  0.21718146, ...,  0.17445193,\n",
       "         0.18694588,  0.27295645],\n",
       "       [ 0.20503877,  0.16525191,  0.13290617, ...,  0.24653084,\n",
       "         0.16014088,  0.11859763],\n",
       "       ...,\n",
       "       [ 0.06784207,  0.05267173,  0.04871502, ..., -0.00293882,\n",
       "         0.05363663,  0.03163469],\n",
       "       [ 0.16169409,  0.05611053,  0.12567384, ...,  0.0716342 ,\n",
       "         0.07090872,  0.10072668],\n",
       "       [ 0.07470506,  0.12240172,  0.06900523, ...,  0.11972307,\n",
       "         0.04162689,  0.04148881]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simMat, wordVectors, wordIdxs, wordNames = item.get_similatity_matrix(vectorModel,centroidVectors)\n",
    "display(Markdown(f'#### The Similarity matrix'))\n",
    "simMat\n",
    "\n",
    "simMat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "#### A dafaframe containing the Similarity Matrix"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gioia</th>\n",
       "      <th>rabbia</th>\n",
       "      <th>sorpresa</th>\n",
       "      <th>disgusto</th>\n",
       "      <th>paura</th>\n",
       "      <th>tristezza</th>\n",
       "      <th>fiducia</th>\n",
       "      <th>attese</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>abate-s</th>\n",
       "      <td>0.035731</td>\n",
       "      <td>0.127025</td>\n",
       "      <td>0.006314</td>\n",
       "      <td>0.038059</td>\n",
       "      <td>0.046227</td>\n",
       "      <td>0.035676</td>\n",
       "      <td>-0.022787</td>\n",
       "      <td>0.047874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>abbaglio-s</th>\n",
       "      <td>0.270037</td>\n",
       "      <td>0.178066</td>\n",
       "      <td>0.217181</td>\n",
       "      <td>0.097142</td>\n",
       "      <td>0.198298</td>\n",
       "      <td>0.174452</td>\n",
       "      <td>0.186946</td>\n",
       "      <td>0.272956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>abbandono-s</th>\n",
       "      <td>0.205039</td>\n",
       "      <td>0.165252</td>\n",
       "      <td>0.132906</td>\n",
       "      <td>0.168139</td>\n",
       "      <td>0.177458</td>\n",
       "      <td>0.246531</td>\n",
       "      <td>0.160141</td>\n",
       "      <td>0.118598</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>abbassamento-s</th>\n",
       "      <td>0.139864</td>\n",
       "      <td>0.151562</td>\n",
       "      <td>0.111983</td>\n",
       "      <td>0.137748</td>\n",
       "      <td>0.151613</td>\n",
       "      <td>0.189760</td>\n",
       "      <td>0.144598</td>\n",
       "      <td>0.215484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>abbattimento-s</th>\n",
       "      <td>0.127449</td>\n",
       "      <td>-0.066286</td>\n",
       "      <td>0.092745</td>\n",
       "      <td>0.081225</td>\n",
       "      <td>0.066279</td>\n",
       "      <td>0.011238</td>\n",
       "      <td>0.062617</td>\n",
       "      <td>0.117573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>zoppo-a</th>\n",
       "      <td>0.044778</td>\n",
       "      <td>0.094590</td>\n",
       "      <td>0.049796</td>\n",
       "      <td>0.152718</td>\n",
       "      <td>0.125573</td>\n",
       "      <td>0.117387</td>\n",
       "      <td>0.058295</td>\n",
       "      <td>0.134320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>zuccherato-a</th>\n",
       "      <td>0.114434</td>\n",
       "      <td>0.084678</td>\n",
       "      <td>0.059379</td>\n",
       "      <td>0.033473</td>\n",
       "      <td>0.038127</td>\n",
       "      <td>0.005117</td>\n",
       "      <td>0.065668</td>\n",
       "      <td>0.040402</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>zuccherino-a</th>\n",
       "      <td>0.067842</td>\n",
       "      <td>0.052672</td>\n",
       "      <td>0.048715</td>\n",
       "      <td>0.036028</td>\n",
       "      <td>0.032904</td>\n",
       "      <td>-0.002939</td>\n",
       "      <td>0.053637</td>\n",
       "      <td>0.031635</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>zuccheroso-a</th>\n",
       "      <td>0.161694</td>\n",
       "      <td>0.056111</td>\n",
       "      <td>0.125674</td>\n",
       "      <td>0.158395</td>\n",
       "      <td>0.075962</td>\n",
       "      <td>0.071634</td>\n",
       "      <td>0.070909</td>\n",
       "      <td>0.100727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>zuppo-a</th>\n",
       "      <td>0.074705</td>\n",
       "      <td>0.122402</td>\n",
       "      <td>0.069005</td>\n",
       "      <td>0.086913</td>\n",
       "      <td>0.104308</td>\n",
       "      <td>0.119723</td>\n",
       "      <td>0.041627</td>\n",
       "      <td>0.041489</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>29992 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   gioia    rabbia  sorpresa  disgusto     paura  tristezza  \\\n",
       "abate-s         0.035731  0.127025  0.006314  0.038059  0.046227   0.035676   \n",
       "abbaglio-s      0.270037  0.178066  0.217181  0.097142  0.198298   0.174452   \n",
       "abbandono-s     0.205039  0.165252  0.132906  0.168139  0.177458   0.246531   \n",
       "abbassamento-s  0.139864  0.151562  0.111983  0.137748  0.151613   0.189760   \n",
       "abbattimento-s  0.127449 -0.066286  0.092745  0.081225  0.066279   0.011238   \n",
       "...                  ...       ...       ...       ...       ...        ...   \n",
       "zoppo-a         0.044778  0.094590  0.049796  0.152718  0.125573   0.117387   \n",
       "zuccherato-a    0.114434  0.084678  0.059379  0.033473  0.038127   0.005117   \n",
       "zuccherino-a    0.067842  0.052672  0.048715  0.036028  0.032904  -0.002939   \n",
       "zuccheroso-a    0.161694  0.056111  0.125674  0.158395  0.075962   0.071634   \n",
       "zuppo-a         0.074705  0.122402  0.069005  0.086913  0.104308   0.119723   \n",
       "\n",
       "                 fiducia    attese  \n",
       "abate-s        -0.022787  0.047874  \n",
       "abbaglio-s      0.186946  0.272956  \n",
       "abbandono-s     0.160141  0.118598  \n",
       "abbassamento-s  0.144598  0.215484  \n",
       "abbattimento-s  0.062617  0.117573  \n",
       "...                  ...       ...  \n",
       "zoppo-a         0.058295  0.134320  \n",
       "zuccherato-a    0.065668  0.040402  \n",
       "zuccherino-a    0.053637  0.031635  \n",
       "zuccheroso-a    0.070909  0.100727  \n",
       "zuppo-a         0.041627  0.041489  \n",
       "\n",
       "[29992 rows x 8 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(simMat, index = wordNames,columns = centroidNames)\n",
    "display(Markdown(f'#### A dafaframe containing the Similarity Matrix'))\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the emotive scores for a single word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Emotive scores for **urlo-s**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "gioia        0.274092\n",
       "rabbia       0.203188\n",
       "sorpresa     0.139861\n",
       "disgusto     0.092078\n",
       "paura        0.130088\n",
       "tristezza    0.165643\n",
       "fiducia      0.155741\n",
       "attese       0.115909\n",
       "Name: urlo-s, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w = 'urlo-s'\n",
    "\n",
    "#For tokenized spaces:\n",
    "#w = 'urlo'\n",
    "\n",
    "display(Markdown(f'Emotive scores for **{w}**'))\n",
    "df.loc[w]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize the data sorted by a single emotion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Dataframe sorted by **gioia**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gioia</th>\n",
       "      <th>rabbia</th>\n",
       "      <th>sorpresa</th>\n",
       "      <th>disgusto</th>\n",
       "      <th>paura</th>\n",
       "      <th>tristezza</th>\n",
       "      <th>fiducia</th>\n",
       "      <th>attese</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>allegro-a</th>\n",
       "      <td>0.672834</td>\n",
       "      <td>0.218464</td>\n",
       "      <td>0.381791</td>\n",
       "      <td>0.238858</td>\n",
       "      <td>0.259723</td>\n",
       "      <td>0.193698</td>\n",
       "      <td>0.350170</td>\n",
       "      <td>0.264201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gioioso-a</th>\n",
       "      <td>0.670783</td>\n",
       "      <td>0.222633</td>\n",
       "      <td>0.411944</td>\n",
       "      <td>0.221171</td>\n",
       "      <td>0.275872</td>\n",
       "      <td>0.264995</td>\n",
       "      <td>0.334075</td>\n",
       "      <td>0.272663</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spensierato-a</th>\n",
       "      <td>0.663996</td>\n",
       "      <td>0.224257</td>\n",
       "      <td>0.357196</td>\n",
       "      <td>0.241643</td>\n",
       "      <td>0.279935</td>\n",
       "      <td>0.260837</td>\n",
       "      <td>0.355128</td>\n",
       "      <td>0.250717</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>entusiasta-a</th>\n",
       "      <td>0.591442</td>\n",
       "      <td>0.388864</td>\n",
       "      <td>0.419100</td>\n",
       "      <td>0.261177</td>\n",
       "      <td>0.387491</td>\n",
       "      <td>0.283486</td>\n",
       "      <td>0.583237</td>\n",
       "      <td>0.361841</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>festoso-a</th>\n",
       "      <td>0.590573</td>\n",
       "      <td>0.169645</td>\n",
       "      <td>0.284783</td>\n",
       "      <td>0.168762</td>\n",
       "      <td>0.209057</td>\n",
       "      <td>0.189748</td>\n",
       "      <td>0.262713</td>\n",
       "      <td>0.191822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aggiungete-v</th>\n",
       "      <td>-0.173104</td>\n",
       "      <td>-0.053790</td>\n",
       "      <td>-0.089117</td>\n",
       "      <td>0.008733</td>\n",
       "      <td>-0.011488</td>\n",
       "      <td>-0.115068</td>\n",
       "      <td>-0.030349</td>\n",
       "      <td>-0.132705</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unire-v</th>\n",
       "      <td>-0.187639</td>\n",
       "      <td>-0.056820</td>\n",
       "      <td>-0.103996</td>\n",
       "      <td>0.028380</td>\n",
       "      <td>-0.015152</td>\n",
       "      <td>-0.132972</td>\n",
       "      <td>-0.055544</td>\n",
       "      <td>-0.132836</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aggiungere-v</th>\n",
       "      <td>-0.210280</td>\n",
       "      <td>-0.067320</td>\n",
       "      <td>-0.114451</td>\n",
       "      <td>0.019027</td>\n",
       "      <td>-0.033565</td>\n",
       "      <td>-0.162857</td>\n",
       "      <td>-0.080599</td>\n",
       "      <td>-0.124044</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tritare-v</th>\n",
       "      <td>-0.217291</td>\n",
       "      <td>-0.058656</td>\n",
       "      <td>-0.144433</td>\n",
       "      <td>0.020543</td>\n",
       "      <td>-0.024580</td>\n",
       "      <td>-0.112895</td>\n",
       "      <td>-0.064962</td>\n",
       "      <td>-0.159197</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>grattugiare-v</th>\n",
       "      <td>-0.234199</td>\n",
       "      <td>-0.057482</td>\n",
       "      <td>-0.126863</td>\n",
       "      <td>0.022036</td>\n",
       "      <td>-0.008152</td>\n",
       "      <td>-0.103996</td>\n",
       "      <td>-0.123058</td>\n",
       "      <td>-0.166486</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>29992 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                  gioia    rabbia  sorpresa  disgusto     paura  tristezza  \\\n",
       "allegro-a      0.672834  0.218464  0.381791  0.238858  0.259723   0.193698   \n",
       "gioioso-a      0.670783  0.222633  0.411944  0.221171  0.275872   0.264995   \n",
       "spensierato-a  0.663996  0.224257  0.357196  0.241643  0.279935   0.260837   \n",
       "entusiasta-a   0.591442  0.388864  0.419100  0.261177  0.387491   0.283486   \n",
       "festoso-a      0.590573  0.169645  0.284783  0.168762  0.209057   0.189748   \n",
       "...                 ...       ...       ...       ...       ...        ...   \n",
       "aggiungete-v  -0.173104 -0.053790 -0.089117  0.008733 -0.011488  -0.115068   \n",
       "unire-v       -0.187639 -0.056820 -0.103996  0.028380 -0.015152  -0.132972   \n",
       "aggiungere-v  -0.210280 -0.067320 -0.114451  0.019027 -0.033565  -0.162857   \n",
       "tritare-v     -0.217291 -0.058656 -0.144433  0.020543 -0.024580  -0.112895   \n",
       "grattugiare-v -0.234199 -0.057482 -0.126863  0.022036 -0.008152  -0.103996   \n",
       "\n",
       "                fiducia    attese  \n",
       "allegro-a      0.350170  0.264201  \n",
       "gioioso-a      0.334075  0.272663  \n",
       "spensierato-a  0.355128  0.250717  \n",
       "entusiasta-a   0.583237  0.361841  \n",
       "festoso-a      0.262713  0.191822  \n",
       "...                 ...       ...  \n",
       "aggiungete-v  -0.030349 -0.132705  \n",
       "unire-v       -0.055544 -0.132836  \n",
       "aggiungere-v  -0.080599 -0.124044  \n",
       "tritare-v     -0.064962 -0.159197  \n",
       "grattugiare-v -0.123058 -0.166486  \n",
       "\n",
       "[29992 rows x 8 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emotion = 'gioia'\n",
    "display(Markdown(f'Dataframe sorted by **{emotion}**'))\n",
    "\n",
    "df.sort_values(by=[emotion],ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering data by emotion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Dataframe by **gioia**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gioia</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>abbaglio-s</th>\n",
       "      <td>0.270037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>abbandono-s</th>\n",
       "      <td>0.205039</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>abbiate-s</th>\n",
       "      <td>0.246437</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>abbondanza-s</th>\n",
       "      <td>0.370326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>abbraccio-s</th>\n",
       "      <td>0.262834</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>volubile-a</th>\n",
       "      <td>0.249351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>voluto-a</th>\n",
       "      <td>0.222829</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>vorticoso-a</th>\n",
       "      <td>0.288817</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>votato-a</th>\n",
       "      <td>0.214935</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>zitto-a</th>\n",
       "      <td>0.222729</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4380 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 gioia\n",
       "abbaglio-s    0.270037\n",
       "abbandono-s   0.205039\n",
       "abbiate-s     0.246437\n",
       "abbondanza-s  0.370326\n",
       "abbraccio-s   0.262834\n",
       "...                ...\n",
       "volubile-a    0.249351\n",
       "voluto-a      0.222829\n",
       "vorticoso-a   0.288817\n",
       "votato-a      0.214935\n",
       "zitto-a       0.222729\n",
       "\n",
       "[4380 rows x 1 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emotion = 'gioia'\n",
    "\n",
    "display(Markdown(f'Dataframe by **{emotion}**'))\n",
    "emo_df = df[emotion]\n",
    "emo_df = pd.DataFrame(emo_df, columns = [emotion])\n",
    "emo_df[emo_df[emotion] >= cosine_threshold]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Writing the output\n",
    "\n",
    "The output is written vita the `item.writeOutput` function. The function takes the following arguments:\n",
    "\n",
    "\n",
    "1. the dataframe containing the emotions, words and cosines `df`\n",
    "2. the path of the output file `outputFile`\n",
    "3. the cosine similarity threshold `cosine_threshold`\n",
    "\n",
    "The function writes the dataframe as a `TAB`-separated CSV. The function returns the output file path as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(simMat, columns = centroidNames)\n",
    "df.insert(0,'word',wordNames)\n",
    "outputFile = item.writeOutput(df, outputFile, cosine_threshold)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Different emotive spaces by PoS (only for PoS-tagged data)\n",
    "\n",
    "For some applications, it may be useful to build the centroid vectors based on the Part-of-Speech. \n",
    "This method was applied in Passaro et al. (2015) in order to construct different emotive centroids according to the PoS (namely Nouns, Verbs, Adjectives), starting from the assumption that the context that best captures the meaning of a word depends on the type of word to be represented. In this case, the emotive scores of a new word are computed on the basis of the cosine similarity between the vector of the target lemma and the centroid vector of the corresponding PoS, i.e. built from the embeddings of the seed lemmas with the same PoS of the target lemma.\n",
    "\n",
    "In this case the functions used to build the emotive space are the same described above, but they are invoked separately by PoS in a for loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "#### Emotive centroids for the Pos S and number of seeds used to build them: \n",
       "\n",
       "[('gioia', 26), ('rabbia', 30), ('sorpresa', 17), ('disgusto', 21), ('paura', 20), ('tristezza', 21), ('fiducia', 21), ('attese', 22)]\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "#### Emotive centroids for the Pos V and number of seeds used to build them: \n",
       "\n",
       "[('gioia', 16), ('rabbia', 15), ('sorpresa', 18), ('disgusto', 21), ('paura', 22), ('tristezza', 23), ('fiducia', 16), ('attese', 23)]\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "#### Emotive centroids for the Pos A and number of seeds used to build them: \n",
       "\n",
       "[('gioia', 19), ('rabbia', 32), ('sorpresa', 25), ('disgusto', 38), ('paura', 36), ('tristezza', 33), ('fiducia', 25), ('attese', 15)]\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "poses = ['s','v','a']\n",
    "\n",
    "for pos in poses:\n",
    "    \n",
    "    posVocab = [w for w in vectorModel.vocab if w.endswith(f'-{pos}')]\n",
    "    posModel = dict()\n",
    "    posCentroids = item.load_seeds(seedsFile)\n",
    "    for w in posVocab:\n",
    "        posModel[w] = vectorModel[w]\n",
    "    posCentroidIdxs, posCentroidNames, posCentroidVectors = item.get_centroid_vectors(posCentroids, posModel)\n",
    "    posCentroids = {key:[a for a in value if a.endswith(f'-{pos}')] for key,value in posCentroids.items()}\n",
    "    display(Markdown(f'#### Emotive centroids for the Pos {pos.upper()} and number of seeds used to build them: \\n\\n{[(k, len(posCentroids[k])) for k in posCentroids]}\\n'))\n",
    "\n",
    "\n",
    "    posSimMat, posWordVectors, posWordIdxs, posWordNames = item.get_similatity_matrix(posModel,posCentroidVectors)\n",
    "    \n",
    "    outputFile = f\"output/ItEM-{pos.upper()}-{cosine_threshold}.cos\"\n",
    "    df = pd.DataFrame(posSimMat, columns = posCentroidNames)\n",
    "    df.insert(0,'word',posWordNames)\n",
    "    outputFile = item.writeOutput(df, outputFile, cosine_threshold)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Differences among the two approaches\n",
    "\n",
    "It is clear that by using non PoS-specific centroids, the cosine similarity between target words and centroids tends to be lower. This is due to the fact that vectors of words with the same PoS are closer to each other in the vector space with respect to words with different PoS. This implies that a centroid built using different PoSes may be less representative of the words due to the sparsity of data points in the high-dimensional vector space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Emotive scores for **spensierato-a** in the _full_ emotive space"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "rabbia       0.224257\n",
       "disgusto     0.241643\n",
       "attese       0.250717\n",
       "tristezza    0.260837\n",
       "paura        0.279935\n",
       "fiducia      0.355128\n",
       "sorpresa     0.357196\n",
       "gioia        0.663996\n",
       "Name: spensierato-a, dtype: float64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "Emotive scores for **spensierato-a** in the _pos-specific_ emotive space"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "disgusto     0.123351\n",
       "rabbia       0.162872\n",
       "paura        0.238634\n",
       "attese       0.249335\n",
       "tristezza    0.280354\n",
       "sorpresa     0.312565\n",
       "fiducia      0.334119\n",
       "gioia        0.787650\n",
       "Name: spensierato-a, dtype: float64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "generalDf = pd.DataFrame(simMat, index = wordNames,columns = centroidNames)\n",
    "posSpecificDf = pd.DataFrame(posSimMat, index = posWordNames,columns = posCentroidNames)\n",
    "\n",
    "w = 'spensierato-a'\n",
    "display(Markdown(f'Emotive scores for **{w}** in the _full_ emotive space'))\n",
    "\n",
    "\n",
    "display(generalDf.loc[w].sort_values())\n",
    "\n",
    "display(Markdown(f'Emotive scores for **{w}** in the _pos-specific_ emotive space'))\n",
    "display(posSpecificDf.loc[w].sort_values())\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "[Passaro et al., (2015): ItEM: A Vector Space Model to Bootstrap an Italian Emotive Lexicon](https://arpi.unipi.it/retrieve/handle/11568/766226/80602/clic-2015-2.pdf)\n",
    "  \n",
    "[Passaro, Lenci (2016): Evaluating Context Selection Strategies to Build Emotive Vector Space Models](http://colinglab.humnet.unipi.it/wp-content/uploads/2012/12/Passaro_Lenci_LREC2016.pdf)\n",
    "\n",
    "[Passaro et al., (2016): FB-NEWS15: A Topic-Annotated Facebook Corpus for Emotion Detection and Sentiment Analysis](http://colinglab.humnet.unipi.it/wp-content/uploads/2012/12/passaro_etal_CLIC2016.pdf)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "item",
   "language": "python",
   "name": "item"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}