3. History of tokens (all_content).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial 3: History of tokens (all_content)\n",
    "\n",
    "Wikiwho allows to track the whole history of all tokens ever written in Wikipedia. In general, one can find all the revisions that inserted or deleted a token. This means that one can track the evolution of the token, e.g. when was the first time was inserted (its revision and its editor, aka author), or how many times the token has been reinserted or removed.\n",
    "\n",
    "The basic conceps of how to interpret the data are given in this section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Querying all content\n",
    "\n",
    "To query all the history of all tokens in a revision, one can use the `all_content()` method as follows:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from wikiwho_wrapper import WikiWho\n",
    "ww = WikiWho(lng='en')\n",
    "df = ww.dv.all_content(article=\"Bioglass 45S5\")\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the explanation of the columns of the above dataframe:\n",
    "\n",
    " - **article_title**: the name of the title\n",
    " - **page_id**: the id of the title\n",
    " - **o_rev_id**: The ID of the revision where the token was added for the first time in the article.\n",
    " - **o_editor**: The user ID of the editor that added the token for the first time (directly related to o_rev_id). User IDs are integers, are unique for the whole Wikipedia and can be used to fetch the current name of a user. \n",
    " - **token**: Actual token value as string. A token can appear multiple times, i.e. it is not unique. This is because a word can be repeated in the document.\n",
    " - **token_id**: The token ID assigned internally by the WikiWho algorithm. The token_id describe a token uniquely because it takes into consideration the context in which the token appear (e.g. paragraph, sentence, position in the sentence). \n",
    " - **in**: revision in which the token has been reinserted\n",
    " - **out**: revision in which the token has been removed\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Use cases for in and out columns\n",
    "\n",
    "The `in` and `out` columns are useful to extract information regarding the history of a token. Here are some use cases:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Multiple editions\n",
    "\n",
    "The token `'sida'` (`token_id=378`) originally inserted in revision `189370281` (the `-1` in column `in` means that it was the original insertion which is indicated in column `o_rev_id` ), deleted in `189370332`, reinserted in `189371159`, deleted again in `189371182`, reinserted again in `189537330`, and finaly deleted in `191585577`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df['token_id'] == 378]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we have used the token_id as opposed to the string of the token (i.e. `'sida'`). One can do that, but one should considered that there might be different tokens with with same name, e.g. `'bioglass'` would out the history of several token ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df['token'] == 'bioglass'].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Reinserted token\n",
    "The token `bioglass` originally inserted in revision `18064039`, deleted in `758323388`, and reinserted in `758323485`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df['token_id'] == 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 Never deleted token\n",
    "The token `is` inserted in revision `18064039` and never taken out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df['token_id'] == 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Basic filters on the dataframe\n",
    "\n",
    "A `-1`  in the `in` column means that there was no reinsertion, i.e. it was a first-time insertion (`o_rev_id`).\n",
    "\n",
    "A `-1` in the `out` column means that the token was not taken out for the corresponding `o_rev_id` or `in` first-time insertion or reinsertion.\n",
    "\n",
    "This can be used to create useful filters:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 (First-time) token insertions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df['in'] == -1].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Token reinsertions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df['in'] != -1].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 Token Deletions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df['out'] != -1].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4 Tokens that still exists in the current revision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df['out'] == -1].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Number of times that each token has been inserted or reinserted\n",
    "\n",
    "Another example is checking the most conflicted tokens, by couted how many times they have been instered or resinserted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sizes = df.groupby(['token_id']).size().sort_values(ascending=False)\n",
    "sizes.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.notebooks import get_next_notebook\n",
    "from IPython.display import HTML\n",
    "try:\n",
    "    display(HTML(f'<a href=\"{get_next_notebook()}\" target=\"_blank\">Go to next workbook</a>'))\n",
    "except:\n",
    "    HTML('<a href=\"4. Token history of specific revision.ipynb\" target=\"_blank\">Go to next workbook</a>')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}