py-gaoya/examples/deduplication_scholarly_articles_gaoya.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "65b151cd",
   "metadata": {},
   "source": [
    "# Document Deduplication with Gaoya"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f12339e",
   "metadata": {},
   "source": [
    "This notebook was based on [Document Deduplication with Similarity Search](https://github.com/pinecone-io/examples/blob/master/deduplication/deduplication_scholarly_articles.ipynb) from Pinecone. The original notebook uses Pinecone vector similarity search to select a set of candidates, which is then further filtered using datasketch MinHash. \n",
    "In this notebook we utilize gaoya MinHashIndex to find near duplicates. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20de5a73",
   "metadata": {},
   "source": [
    "First, install the following dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e592330e",
   "metadata": {},
   "source": [
    "<code>pip install gaoya</code>\n",
    "\n",
    "<code>pip install datasets</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "7ff47b1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import math\n",
    "import statistics\n",
    "import gaoya\n",
    "from datasets import load_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "610f940b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.1.2'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gaoya.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91dba9ff",
   "metadata": {},
   "source": [
    "In this tutorial, we will use the [Deduplication Dataset 2020](https://core.ac.uk/documentation/dataset) that consists of 100,000 scholarly documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "bf0fa807",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration pinecone--core-2020-05-10-deduplication-135d0b69d52bf68c\n",
      "Reusing dataset json (/home/sergey/.cache/huggingface/datasets/pinecone___json/pinecone--core-2020-05-10-deduplication-135d0b69d52bf68c/0.0.0/da492aad5680612e4028e7f6ddc04b1dfcec4b64db470ed7cc5f2bb265b9b6b5)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['core_id', 'doi', 'original_abstract', 'original_title', 'processed_title', 'processed_abstract', 'cat', 'labelled_duplicates'],\n",
       "    num_rows: 100000\n",
       "})"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "core = load_dataset(\"pinecone/core-2020-05-10-deduplication\", split=\"train\")\n",
    "core"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9339daab",
   "metadata": {},
   "source": [
    "We convert the dataset into Pandas dataframe format like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "de0500f6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>core_id</th>\n",
       "      <th>doi</th>\n",
       "      <th>original_abstract</th>\n",
       "      <th>original_title</th>\n",
       "      <th>processed_title</th>\n",
       "      <th>processed_abstract</th>\n",
       "      <th>cat</th>\n",
       "      <th>labelled_duplicates</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11251086</td>\n",
       "      <td>10.1016/j.ajhg.2007.12.013</td>\n",
       "      <td>Unobstructed vision requires a particular refr...</td>\n",
       "      <td>Mutation of solute carrier SLC16A12 associates...</td>\n",
       "      <td>mutation of solute carrier slc16a12 associates...</td>\n",
       "      <td>unobstructed vision refractive lens differenti...</td>\n",
       "      <td>exact_dup</td>\n",
       "      <td>[82332306]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11309751</td>\n",
       "      <td>10.1103/PhysRevLett.101.193002</td>\n",
       "      <td>Two-color multiphoton ionization of atomic hel...</td>\n",
       "      <td>Polarization control in two-color above-thresh...</td>\n",
       "      <td>polarization control in two-color above-thresh...</td>\n",
       "      <td>multiphoton ionization helium combining extrem...</td>\n",
       "      <td>exact_dup</td>\n",
       "      <td>[147599753]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11311385</td>\n",
       "      <td>10.1016/j.ab.2011.02.013</td>\n",
       "      <td>Lectin’s are proteins capable of recognising a...</td>\n",
       "      <td>Optimisation of the enzyme-linked lectin assay...</td>\n",
       "      <td>optimisation of the enzyme-linked lectin assay...</td>\n",
       "      <td>lectin’s capable recognising oligosaccharide t...</td>\n",
       "      <td>exact_dup</td>\n",
       "      <td>[147603441]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    core_id                             doi  \\\n",
       "0  11251086      10.1016/j.ajhg.2007.12.013   \n",
       "1  11309751  10.1103/PhysRevLett.101.193002   \n",
       "2  11311385        10.1016/j.ab.2011.02.013   \n",
       "\n",
       "                                   original_abstract  \\\n",
       "0  Unobstructed vision requires a particular refr...   \n",
       "1  Two-color multiphoton ionization of atomic hel...   \n",
       "2  Lectin’s are proteins capable of recognising a...   \n",
       "\n",
       "                                      original_title  \\\n",
       "0  Mutation of solute carrier SLC16A12 associates...   \n",
       "1  Polarization control in two-color above-thresh...   \n",
       "2  Optimisation of the enzyme-linked lectin assay...   \n",
       "\n",
       "                                     processed_title  \\\n",
       "0  mutation of solute carrier slc16a12 associates...   \n",
       "1  polarization control in two-color above-thresh...   \n",
       "2  optimisation of the enzyme-linked lectin assay...   \n",
       "\n",
       "                                  processed_abstract        cat  \\\n",
       "0  unobstructed vision refractive lens differenti...  exact_dup   \n",
       "1  multiphoton ionization helium combining extrem...  exact_dup   \n",
       "2  lectin’s capable recognising oligosaccharide t...  exact_dup   \n",
       "\n",
       "  labelled_duplicates  \n",
       "0          [82332306]  \n",
       "1         [147599753]  \n",
       "2         [147603441]  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = core.to_pandas()\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55ac2a78",
   "metadata": {},
   "source": [
    "Now let us look at the columns in the dataset that are relevant for our task.\n",
    "\n",
    "**core_id** - Unique indentifier for each article\n",
    "\n",
    "**processed_abstract** - This is obtained by applying preprocssing steps like [this](https://spacy.io/usage/processing-pipelines) to the original abstract of the article from the column **original abstract**.\n",
    "\n",
    "**processed_title** - Same as the abstract but for the title of the article.\n",
    "\n",
    "**cat** - Every article falls into one of the three possible categories: 'exact_dup','near_dup','non_dup'\n",
    "\n",
    "**labelled_duplicates** - A list of core_ids of articles that are duplicates of current article\n",
    "\n",
    "Let's calculate the frequency of duplicates per article. Observe that half of the articles have no duplicates, and only a small fraction of the articles have more than ten duplicates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "d315aaaa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     50000\n",
       "1     36166\n",
       "2      7620\n",
       "3      3108\n",
       "4      1370\n",
       "5       756\n",
       "6       441\n",
       "7       216\n",
       "8       108\n",
       "10       66\n",
       "9        60\n",
       "11       48\n",
       "13       28\n",
       "12       13\n",
       "Name: labelled_duplicates, dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lens = df.labelled_duplicates.apply(len)\n",
    "lens.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1f4e5ee",
   "metadata": {},
   "source": [
    "We will make use of the text data to create vectors for every article. We combine the **processed_abstract** and **processed_title** of the article to create a new **combined_text** column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "e8af0629",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a new column for calculating embeddings\n",
    "df[\"combined_text\"] = df.apply(\n",
    "    lambda x: str(x.processed_title) + \" \" + str(x.processed_abstract), axis=1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e054df2",
   "metadata": {},
   "source": [
    "We'll use **MinHashStringindex** from gaoya to minhash everydocument and index them for fast similarity search. Gaoya is implemented in Rust, which is strongly typed compiled language, and types need to be specified at compile time. py-gaoya provides MinHashimplementations for string data with integer ids. \n",
    "We convert **core_id** to int and store in a new column with the same name. We do the same for **labeled_duplicates**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "6dc06714",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['core_id'] = df['core_id'].astype(int)\n",
    "df['labelled_duplicates'] = df.labelled_duplicates.apply(lambda x: [int(i) for i in x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21628ea9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "31187454",
   "metadata": {},
   "source": [
    "To create MinHash strings needs to be tokenized into words or shingles. Gaoya provides high performance tokenizers, but allows clients to choose any tokenization scheme. \n",
    "Here we are using 3-4 character ngrams ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "d862460b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def _analyzer(doc): return doc.split()\n",
    "minhash_index = gaoya.minhash.MinHashStringIndex(hash_size=32, jaccard_threshold=0.5, num_bands=50, band_size=4, num_hashes=None, analyzer='char', lowercase=False, ngram_range=(3, 4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "cd8885a1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MinHashIndex<u32> { threshold = 0.5, num_hashes = 200, bands = 50, rows_per_band = 4, size = 0 } CharShingle((3, Some(4)))"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "minhash_index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0f9b425",
   "metadata": {},
   "source": [
    "We could insert data using a loop one document at a time. Instead, we call the method `par_bulk_insert_docs` that uses multiple cores to insert data into index in parallel. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "1c179232",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 30.5 s, sys: 307 ms, total: 30.8 s\n",
      "Wall time: 2.66 s\n"
     ]
    }
   ],
   "source": [
    "%time minhash_index.par_bulk_insert_docs(list(df['core_id']), list(df['combined_text']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "45b08a9f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>core_id</th>\n",
       "      <th>doi</th>\n",
       "      <th>original_abstract</th>\n",
       "      <th>original_title</th>\n",
       "      <th>processed_title</th>\n",
       "      <th>processed_abstract</th>\n",
       "      <th>cat</th>\n",
       "      <th>labelled_duplicates</th>\n",
       "      <th>combined_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>30000</th>\n",
       "      <td>33751947</td>\n",
       "      <td>10.1007/s10955-013-0720-1</td>\n",
       "      <td>In this paper we present a novel method to rec...</td>\n",
       "      <td>Bootstrapping Topological Properties and Syste...</td>\n",
       "      <td>bootstrapping topological properties and syste...</td>\n",
       "      <td>reconstruct topological information. topologic...</td>\n",
       "      <td>near_exact_dup</td>\n",
       "      <td>[24767943]</td>\n",
       "      <td>bootstrapping topological properties and syste...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30001</th>\n",
       "      <td>34623027</td>\n",
       "      <td>10.1007/s10260-015-0297-8</td>\n",
       "      <td>Functional data are occurring more and more of...</td>\n",
       "      <td>Multivariate functional outlier detection</td>\n",
       "      <td>multivariate functional outlier detection</td>\n",
       "      <td>occurring analyze them. multivariate observed....</td>\n",
       "      <td>near_exact_dup</td>\n",
       "      <td>[29525269]</td>\n",
       "      <td>multivariate functional outlier detection occu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30002</th>\n",
       "      <td>34648486</td>\n",
       "      <td>10.1016/j.physletb.2015.11.047</td>\n",
       "      <td>We thank James de Boer and Karsten Riisager fo...</td>\n",
       "      <td>Measurement of the branching ratio for β-delay...</td>\n",
       "      <td>measurement of the branching ratio for β-delay...</td>\n",
       "      <td>james boer karsten riisager helpful cussion an...</td>\n",
       "      <td>near_exact_dup</td>\n",
       "      <td>[29549700, 81209193]</td>\n",
       "      <td>measurement of the branching ratio for β-delay...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30003</th>\n",
       "      <td>35079441</td>\n",
       "      <td>10.1016/j.physletb.2014.01.063</td>\n",
       "      <td>We report the first experimental upper bound t...</td>\n",
       "      <td>A first experimental limit on in-matter torsio...</td>\n",
       "      <td>a first experimental limit on in-matter torsio...</td>\n",
       "      <td>torsion neutron parity violation neutron const...</td>\n",
       "      <td>near_exact_dup</td>\n",
       "      <td>[24976215, 81218577]</td>\n",
       "      <td>a first experimental limit on in-matter torsio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30004</th>\n",
       "      <td>35089352</td>\n",
       "      <td>10.1007/JHEP06(2015)175</td>\n",
       "      <td>We study the η - η ′ mixing up to next-to-next...</td>\n",
       "      <td>Scrutinizing the η - η ′ mixing, masses and ps...</td>\n",
       "      <td>scrutinizing the η - η ′ mixing, masses and ps...</td>\n",
       "      <td>chiral perturbation phenomenological inputs. o...</td>\n",
       "      <td>near_exact_dup</td>\n",
       "      <td>[35089431, 60653719, 87082462]</td>\n",
       "      <td>scrutinizing the η - η ′ mixing, masses and ps...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        core_id                             doi  \\\n",
       "30000  33751947       10.1007/s10955-013-0720-1   \n",
       "30001  34623027       10.1007/s10260-015-0297-8   \n",
       "30002  34648486  10.1016/j.physletb.2015.11.047   \n",
       "30003  35079441  10.1016/j.physletb.2014.01.063   \n",
       "30004  35089352         10.1007/JHEP06(2015)175   \n",
       "\n",
       "                                       original_abstract  \\\n",
       "30000  In this paper we present a novel method to rec...   \n",
       "30001  Functional data are occurring more and more of...   \n",
       "30002  We thank James de Boer and Karsten Riisager fo...   \n",
       "30003  We report the first experimental upper bound t...   \n",
       "30004  We study the η - η ′ mixing up to next-to-next...   \n",
       "\n",
       "                                          original_title  \\\n",
       "30000  Bootstrapping Topological Properties and Syste...   \n",
       "30001          Multivariate functional outlier detection   \n",
       "30002  Measurement of the branching ratio for β-delay...   \n",
       "30003  A first experimental limit on in-matter torsio...   \n",
       "30004  Scrutinizing the η - η ′ mixing, masses and ps...   \n",
       "\n",
       "                                         processed_title  \\\n",
       "30000  bootstrapping topological properties and syste...   \n",
       "30001          multivariate functional outlier detection   \n",
       "30002  measurement of the branching ratio for β-delay...   \n",
       "30003  a first experimental limit on in-matter torsio...   \n",
       "30004  scrutinizing the η - η ′ mixing, masses and ps...   \n",
       "\n",
       "                                      processed_abstract             cat  \\\n",
       "30000  reconstruct topological information. topologic...  near_exact_dup   \n",
       "30001  occurring analyze them. multivariate observed....  near_exact_dup   \n",
       "30002  james boer karsten riisager helpful cussion an...  near_exact_dup   \n",
       "30003  torsion neutron parity violation neutron const...  near_exact_dup   \n",
       "30004  chiral perturbation phenomenological inputs. o...  near_exact_dup   \n",
       "\n",
       "                  labelled_duplicates  \\\n",
       "30000                      [24767943]   \n",
       "30001                      [29525269]   \n",
       "30002            [29549700, 81209193]   \n",
       "30003            [24976215, 81218577]   \n",
       "30004  [35089431, 60653719, 87082462]   \n",
       "\n",
       "                                           combined_text  \n",
       "30000  bootstrapping topological properties and syste...  \n",
       "30001  multivariate functional outlier detection occu...  \n",
       "30002  measurement of the branching ratio for β-delay...  \n",
       "30003  a first experimental limit on in-matter torsio...  \n",
       "30004  scrutinizing the η - η ′ mixing, masses and ps...  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[30000:30005]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07f8735c",
   "metadata": {},
   "source": [
    "Let's run a query against the index for a one article."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "2912c45c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[87082462, 35089431, 35089352, 60653719]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "minhash_index.query(df.iloc[30004].combined_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5b3d7cf",
   "metadata": {},
   "source": [
    "Let's compare it with actual labelled_duplicates. Note, that the `query` returns also the id of the query document, where **labelled_duplicates** only contains duplicates, so there will always be an extra id in the result set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "2907262c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[35089431, 60653719, 87082462]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[30004].labelled_duplicates"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5228ec73",
   "metadata": {},
   "source": [
    "For every article in the dataset we query the index and store the result in the column **predicted_duplicates**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "72992a97",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 41 s, sys: 183 ms, total: 41.2 s\n",
      "Wall time: 3.61 s\n"
     ]
    }
   ],
   "source": [
    "%time df['predicted_duplicates'] = minhash_index.par_bulk_query(list(df.combined_text.values))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fce7100",
   "metadata": {},
   "source": [
    "Remove the id of the query article for every row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "79c367c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = df.apply(lambda row: row['predicted_duplicates'].remove(row['core_id']), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66fde33e",
   "metadata": {},
   "source": [
    "Let's evaluate the quality of deduplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "be9a0cb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Correct'] = df.apply(lambda row: set(row['labelled_duplicates']) == set(row['predicted_duplicates']), axis=1).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "bca236ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction_summary = { 'Correct' : df['Correct'].sum(), 'Incorrect' : df.shape[0] - df['Correct'].sum() }\n",
    "prediction_summary['Accuracy'] = round(prediction_summary['Correct'] / df.shape[0], 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "de1d5b0b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Correct': 93807, 'Incorrect': 6193, 'Accuracy': 0.9381}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prediction_summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd528e73",
   "metadata": {},
   "source": [
    "We also calculate recall and precision of the deduplication "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "e613067f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def _recall(row):\n",
    "    labelled_dups = set(row['labelled_duplicates'])\n",
    "    if len(labelled_dups) == 0:\n",
    "        return 1\n",
    "    dups = set(row['predicted_duplicates'])\n",
    "    return len(dups & labelled_dups) / len(labelled_dups)\n",
    "recalls = df.apply(lambda row: _recall(row), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "c3f49672",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9623026827616826"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recalls.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "2261c966",
   "metadata": {},
   "outputs": [],
   "source": [
    "def _precision(row):\n",
    "    labelled_dups = set(row['labelled_duplicates'])\n",
    "    dups = set(row['predicted_duplicates'])    \n",
    "    if len(dups) == 0:\n",
    "        return 0\n",
    "\n",
    "    return len(dups & labelled_dups) / len(dups)\n",
    "precisions = df.apply(lambda row: _precision(row), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "53a9b23d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4642050719943295"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precisions.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d800959",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this notebook we demonstrate how to perform a deduplication task of over 100,000 articles using Gaoya. High performance MinHash algorithm implemented in  Rust  allows deduplicate 100K dataset in just a few seconds\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}