notebooks/Project and Developer Similarity.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project and Developer Similarity\n",
    "\n",
    "In this notebook, we study what projects and developers are about and how to find projects or devs that are close to them.\n",
    "\n",
    "We use [Topic Modeling](https://en.wikipedia.org/wiki/Topic_model), through the excellent [BigARTM](http://docs.bigartm.org/en/stable/index.html) library to achieve that. Roughly speaking, the topic model we use sees each code file as stemming from some topics (e.g., `setup.py` might come from topics about packaging and documentation).\n",
    "\n",
    "To be able to apply this topic modeling technique, we need to transform each code file into a bag of identifiers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by defining the paths we'll use for our inputs and outputs later in this notebook.\n",
    "\n",
    "A note on how we use the cells in this notebook: __all cells should be only dependent from the first one (that defines paths)__. It means we will \n",
    "save and load all results in files to achieve that. This helps mitigate the problems that arise from stateful notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from enum import Enum\n",
    "from os import makedirs\n",
    "from os.path import join as path_join\n",
    "from typing import Union\n",
    "\n",
    "from utils import DirsABC, FilesABC, Run\n",
    "\n",
    "class Files(FilesABC, Enum):\n",
    "    IDENTIFIERS = [\"identifiers.jsonl.bz2\"]\n",
    "    SPLIT_IDENTIFIERS = [\"split-identifiers.jsonl.bz2\"]\n",
    "    FILTERED_IDENTIFIERS = [\"filtered-identifiers.jsonl.bz2\"]\n",
    "    IDENTIFIERS_COUNTER = [\"identifiers-counter.pickle\"]\n",
    "    COMMON_IDENTIFIERS_COUNTER = [\"common-identifiers-counter.pickle\"]\n",
    "    VW_DATASET = [\"dataset.vw\"]\n",
    "    ARTM_DICT = [\"bigartm\", \"identifiers.dict\"]\n",
    "    ARTM_STAGE1 = [\"bigartm\", \"stage1.model\"]\n",
    "    ARTM_STAGE2 = [\"bigartm\", \"stage2.model\"]\n",
    "    ARTM_FILES_TOPICS = [\"bigartm\", \"files-topics.bigartm\"]\n",
    "    ARTM_TOPICS_IDENTIFIERS = [\"bigartm\", \"topics-identifiers.bigartm\"]\n",
    "    PYLDAVIS_DATA = [\"pyldavis-data.pickle\"]\n",
    "    CONTRIBUTIONS = [\"contributions.pickle\"]\n",
    "    REPOS_TOPICS = [\"repos.pickle\"]\n",
    "    AUTHORS_TOPICS = [\"authors.pickle\"]\n",
    "\n",
    "\n",
    "class Dirs(DirsABC, Enum):\n",
    "    ARTM_LOGS = [\"bigartm\", \"logs\"]\n",
    "    ARTM_BATCHES = [\"bigartm\", \"batches\"]\n",
    "\n",
    "\n",
    "full_run = Run(\"similarity\", \"full\")\n",
    "limited_run = Run(\"similarity\", \"limited\")\n",
    "run = limited_run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by the bulk of the preprocessing: extracting identifiers from code with [`gitbase`](http://docs.bigartm.org/en/stable/index.html). `gitbase` exposes git repositories as SQL databases with the following schema:\n",
    "\n",
    "![`gitbase` schema](img/gitbase-schema.png)\n",
    "\n",
    "Ok, it's probably hard to read. It's likely possible to use `Right click` > `View image` to explore it, but we'll extract the tables we're going to use below to make our life easy:\n",
    "\n",
    "![tables](img/tables.png)\n",
    "\n",
    "Using those 3 tables, we can get identifiers with the `uast_extract(blob, key) text array` [`gitbase` function](https://docs.sourced.tech/gitbase/using-gitbase/functions), that leverages [Babelfish](https://doc.bblf.sh/). The nice point about using Babelfish is that since it exposes the same API for different languages, we write a query once, and we get a preprocessing that works readily for plenty of languages: c#, c++, c, cuda, opencl, metal, bash, shell, go, java, javascript, jsx, php, python, ruby and typescript.\n",
    "\n",
    "All we have to do now is to write the query!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bz2 import open as bz2_open\n",
    "from json import dumps as json_dumps, loads as json_loads\n",
    "from pprint import pprint\n",
    "\n",
    "from utils import SUPPORTED_LANGUAGES, query_gitbase\n",
    "\n",
    "\n",
    "def extract_identifiers(identifiers_path: str, limit: int = 0):\n",
    "    sql = \"\"\"\n",
    "        SELECT\n",
    "            repository_id,\n",
    "            LANGUAGE(file_path) AS lang,\n",
    "            file_path,\n",
    "            uast_extract(\n",
    "                uast(blob_content,\n",
    "                     LANGUAGE(file_path),\n",
    "                     '//uast:Identifier'),\n",
    "                'Name'\n",
    "            ) AS identifiers\n",
    "        FROM refs\n",
    "        NATURAL JOIN commit_files\n",
    "        NATURAL JOIN blobs\n",
    "        WHERE\n",
    "            ref_name = 'HEAD'\n",
    "            AND NOT IS_VENDOR(file_path)\n",
    "            AND NOT IS_BINARY(file_path)\n",
    "            AND LANGUAGE(file_path) IN (%s)\n",
    "        %s\n",
    "    \"\"\" % (\n",
    "        \",\".join(\"'%s'\" % language for language in SUPPORTED_LANGUAGES),\n",
    "        \"LIMIT %d\" % limit if limit > 0 else \"\"\n",
    "    )\n",
    "    print(\"Extracting identifiers with the following gitbase query:\")\n",
    "    print(sql)\n",
    "    print(\"First extracted rows:\")\n",
    "    with bz2_open(identifiers_path, \"wt\", encoding=\"utf8\") as fh:\n",
    "        shown = 0\n",
    "        for row in query_gitbase(sql):\n",
    "            if row[\"identifiers\"] is None:\n",
    "                continue\n",
    "            row[\"identifiers\"] = json_loads(row[\"identifiers\"])\n",
    "            while shown < 10:\n",
    "                shown += 1\n",
    "                print(\"Row %d:\" % shown)\n",
    "                pprint(row)\n",
    "            fh.write(\"%s\\n\" % json_dumps(row))\n",
    "\n",
    "\n",
    "extract_identifiers(run.path(Files.IDENTIFIERS), 1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a file that stores all the identifiers, we can refine it until it is ready for topic modeling! The remaining steps are to further split each identifier (`set_timer` should become `set` and `timer`), and to apply some stemming (`connecting` and `connection` should both result in `connect`, note that the result stem might not be an English word)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from bz2 import open as bz2_open\n",
    "from collections import Counter\n",
    "from json import dumps as json_dumps, loads as json_loads\n",
    "from pickle import dump as pickle_dump\n",
    "\n",
    "from utils import TokenParser\n",
    "\n",
    "\n",
    "def split_identifiers(identifiers_path: str,\n",
    "                      split_identifiers_path: str,\n",
    "                      counter_path: str):\n",
    "    with bz2_open(identifiers_path, \"rt\", encoding=\"utf8\") as fh_identifiers, \\\n",
    "            bz2_open(split_identifiers_path, \"wt\", encoding=\"utf8\") as fh_split_identifiers, \\\n",
    "            open(counter_path, \"wb\") as fh_counter:\n",
    "        identifiers_counter = Counter()\n",
    "        token_parser = TokenParser()\n",
    "        shown = set()\n",
    "        print(\"10 first splits:\")\n",
    "        for row_str in fh_identifiers:\n",
    "            row = json_loads(row_str)\n",
    "            identifiers = row.pop(\"identifiers\")\n",
    "            split_identifiers = []\n",
    "            for identifier in identifiers:\n",
    "                split_identifier = list(token_parser(identifier))\n",
    "                split_identifiers.extend(split_identifier)\n",
    "                if (len(shown) < 10\n",
    "                        and identifier not in shown\n",
    "                        and len(split_identifier) > 1):\n",
    "                    shown.add(identifier)\n",
    "                    print(\"Splitting %s into (%s)\" % (\n",
    "                        identifier,\n",
    "                        \", \".join(split_identifier)\n",
    "                    ))\n",
    "            identifiers_counter.update(split_identifiers)\n",
    "            row[\"split_identifiers\"] = split_identifiers\n",
    "            fh_split_identifiers.write(\"%s\\n\" % json_dumps(row))\n",
    "        pickle_dump(identifiers_counter, fh_counter)\n",
    "\n",
    "\n",
    "split_identifiers(run.path(Files.IDENTIFIERS),\n",
    "                  run.path(Files.SPLIT_IDENTIFIERS),\n",
    "                  run.path(Files.IDENTIFIERS_COUNTER))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting identifiers still need some processing: some of them appear only a few times and will bring mostly noise to our models. We will discard them now. The first step is to find out which identifiers are common enough to be kept."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pickle import dump as pickle_dump, load as pickle_load\n",
    "\n",
    "\n",
    "def build_common_counter(count_threshold: int,\n",
    "                         counter_path: str,\n",
    "                         common_counter_path: str):\n",
    "    with open(counter_path, \"rb\") as fh:\n",
    "        identifiers_counter = pickle_load(fh)\n",
    "    print(\"Found %d different identifiers\" % len(identifiers_counter))\n",
    "\n",
    "    common_identifiers_counter = identifiers_counter.copy()\n",
    "    for identifier, count in identifiers_counter.items():\n",
    "        if count < count_threshold:\n",
    "            del common_identifiers_counter[identifier]\n",
    "    with open(common_counter_path, \"wb\") as fh:\n",
    "        pickle_dump(common_identifiers_counter, fh)\n",
    "    print(\"Found %d different identifiers after pruning\"\n",
    "          % len(common_identifiers_counter))\n",
    "\n",
    "\n",
    "build_common_counter(10,\n",
    "                     run.path(Files.IDENTIFIERS_COUNTER),\n",
    "                     run.path(Files.COMMON_IDENTIFIERS_COUNTER))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know which are the common identifiers, we can recreate our mapping from files to identifiers with only the ones that we want to keep."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from json import dumps as json_dumps, loads as json_loads\n",
    "from pickle import load as pickle_load\n",
    "\n",
    "\n",
    "def filter_identifiers(split_identifiers_path: str,\n",
    "                       common_counter_path: str,\n",
    "                       filtered_identifiers_path: str):\n",
    "    with bz2_open(split_identifiers_path, \"rt\", encoding=\"utf8\") as fh_split_identifiers, \\\n",
    "            open(common_counter_path, \"rb\") as fh_common_counter, \\\n",
    "            bz2_open(filtered_identifiers_path, \"wt\", encoding=\"utf8\") as fh_filtered_identifiers:\n",
    "        common_identifiers_counter = pickle_load(fh_common_counter)\n",
    "        for row_str in fh_split_identifiers:\n",
    "            row = json_loads(row_str)\n",
    "            row[\"split_identifiers\"] = [identifier\n",
    "                                        for identifier in row[\"split_identifiers\"]\n",
    "                                        if identifier in common_identifiers_counter]\n",
    "            if row[\"split_identifiers\"]:\n",
    "                fh_filtered_identifiers.write(\"%s\\n\" % json_dumps(row))\n",
    "\n",
    "\n",
    "filter_identifiers(run.path(Files.SPLIT_IDENTIFIERS),\n",
    "                   run.path(Files.COMMON_IDENTIFIERS_COUNTER),\n",
    "                   run.path(Files.FILTERED_IDENTIFIERS))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Topic Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The preprocessing is over! We now create the input dataset, in the VW format (see https://bigartm.readthedocs.io/en/stable/tutorials/datasets.html). We replace spaces in `file_path` to avoid creating false identifiers (VW would consider the latter parts of a path containing spaces to be identifiers)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_file_id(repository_id: str, lang: str, file_path: str):\n",
    "    return \"%s//%s//%s\" % (repository_id,\n",
    "                           lang,\n",
    "                           file_path.replace(\" \", \"_\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "from os.path import join as path_join\n",
    "\n",
    "\n",
    "def build_vw_dataset(filtered_identifiers_path: str,\n",
    "                     vw_dataset_path: str):\n",
    "    !rm -rf vw_dataset_path\n",
    "    with bz2_open(filtered_identifiers_path, \"rt\", encoding=\"utf8\") as fh_filtered_identifiers, \\\n",
    "            open(vw_dataset_path, \"w\") as fh_vw:\n",
    "        shown = 0\n",
    "        print(\"Showing first 10 lines:\")\n",
    "        for row_str in fh_filtered_identifiers:\n",
    "            counter = Counter()\n",
    "            row = json_loads(row_str)\n",
    "            counter.update(row[\"split_identifiers\"])\n",
    "            line = \"%s %s\" % (\n",
    "                build_file_id(row[\"repository_id\"],\n",
    "                              row[\"lang\"],\n",
    "                              row[\"file_path\"]),\n",
    "                \" \".join(\"%s:%d\" % (identifier, count)\n",
    "                     for identifier, count in counter.items())\n",
    "            )\n",
    "            if shown < 10:\n",
    "                shown +=1\n",
    "                print(\"Line %d: %s\" % (shown, line))\n",
    "            fh_vw.write(\"%s\\n\" % line)\n",
    "\n",
    "\n",
    "build_vw_dataset(run.path(Files.FILTERED_IDENTIFIERS),\n",
    "                 run.path(Files.VW_DATASET))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bigartm has its own binary format to efficiently store and access the data used to build its topic models. The next step is therefore to transform our VW dataset into the correct Bigartm format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_bigartm(vw_dataset_path: str,\n",
    "                    artm_batches_path: str,\n",
    "                    artm_dict_path: str,\n",
    "                    artm_logs_path: str):\n",
    "    !rm -rf {artm_batches_path} {artm_dict_path}\n",
    "    !bigartm \\\n",
    "        --log-dir {artm_logs_path} \\\n",
    "        -c {vw_dataset_path} \\\n",
    "        -p 0 \\\n",
    "        --save-batches {artm_batches_path} \\\n",
    "        --save-dictionary {artm_dict_path}\n",
    "\n",
    "\n",
    "prepare_bigartm(run.path(Files.VW_DATASET),\n",
    "                run.path(Dirs.ARTM_BATCHES),\n",
    "                run.path(Files.ARTM_DICT),\n",
    "                run.path(Dirs.ARTM_LOGS))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now train our first topic model! As per the Bigartm documentation, we don't use too much magic (yet), and only use one regularizer --- the decorrelation one. It will make sure that no 2 topics are about the same concepts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from multiprocessing import cpu_count\n",
    "\n",
    "\n",
    "def train_topic_model(artm_batches_path: str,\n",
    "                      artm_dict_path: str,\n",
    "                      artm_stage1_path: str,\n",
    "                      artm_logs_path: str,\n",
    "                      n_topics: int = 64,\n",
    "                      n_epochs: int = 100,\n",
    "                      n_cpus: int = cpu_count() * 2,\n",
    "                      seed: int = 2019,\n",
    "                      regularizer: str = '\"1000 Decorrelation\"'):\n",
    "    !bigartm \\\n",
    "        --log-dir {artm_logs_path} \\\n",
    "        --use-batches {artm_batches_path} \\\n",
    "        --use-dictionary {artm_dict_path} \\\n",
    "        -t {n_topics} \\\n",
    "        -p {n_epochs} \\\n",
    "        --threads {n_cpus} \\\n",
    "        --rand-seed {seed} \\\n",
    "        --regularizer {regularizer} \\\n",
    "        --save-model {artm_stage1_path} \\\n",
    "        --force\n",
    "\n",
    "\n",
    "train_topic_model(run.path(Dirs.ARTM_BATCHES),\n",
    "                  run.path(Files.ARTM_DICT),\n",
    "                  run.path(Files.ARTM_STAGE1),\n",
    "                  run.path(Dirs.ARTM_LOGS))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This topic model is probably quite good already, but since Bigartm is a powerful library, we can improve it even further by making it sparser. Sparse topics for a document means that it will have mainly a few topics with high weight and other topics with weight 0.\n",
    "\n",
    "Example of non-sparse documents:\n",
    "\n",
    "|           | Backend | Logging | Machine Learning | Data Processing |\n",
    "|-----------|---------|---------|------------------|-----------------|\n",
    "| server.py | 0.8     | 0.1     | 0.06             | 0.04            |\n",
    "| utils.py  | 0.1     | 0.7     | 0.08             | 0.12            |\n",
    "\n",
    "Same documents but with sparse topics:\n",
    "\n",
    "|           | Backend | Logging | Machine Learning | Data Processing |\n",
    "|-----------|---------|---------|------------------|-----------------|\n",
    "| server.py | 0.85    | 0.15    | 0                | 0               |\n",
    "| utils.py  | 0.12    | 0.88    | 0                | 0               |\n",
    "\n",
    "This makes understanding the documents and topics easier: they contain less non-zero entries and are more focused on the most important stuff."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sparsify_topic_model(\n",
    "    artm_batches_path: str,\n",
    "    artm_dict_path: str,\n",
    "    artm_stage1_path: str,\n",
    "    artm_stage2_path: str,\n",
    "    artm_files_topics_path: str,\n",
    "    artm_topics_identifiers_path: str,\n",
    "    artm_logs_path: str,\n",
    "    n_epochs: int = 20,\n",
    "    n_cpus: int = cpu_count() * 2,\n",
    "    seed: int = 2019,\n",
    "    regularizer: str = ' \"1000 Decorrelation\" \"0.5 SparsePhi\" \"0.5 SparseTheta\" '\n",
    "):\n",
    "    !bigartm \\\n",
    "        --log-dir {artm_logs_path} \\\n",
    "        --use-batches {artm_batches_path} \\\n",
    "        --use-dictionary {artm_dict_path} \\\n",
    "        --load-model {artm_stage1_path} \\\n",
    "        -p {n_epochs} \\\n",
    "        --threads {n_cpus} \\\n",
    "        --rand-seed {seed} \\\n",
    "        --regularizer {regularizer} \\\n",
    "        --save-model {artm_stage2_path} \\\n",
    "        --force \\\n",
    "        --write-predictions {artm_files_topics_path} \\\n",
    "        --write-model-readable {artm_topics_identifiers_path}\n",
    "\n",
    "\n",
    "sparsify_topic_model(run.path(Dirs.ARTM_BATCHES),\n",
    "                     run.path(Files.ARTM_DICT),\n",
    "                     run.path(Files.ARTM_STAGE1),\n",
    "                     run.path(Files.ARTM_STAGE2),\n",
    "                     run.path(Files.ARTM_FILES_TOPICS),\n",
    "                     run.path(Files.ARTM_TOPICS_IDENTIFIERS),\n",
    "                     run.path(Dirs.ARTM_LOGS))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our topic model should be perfectly cooked now. It's time to taste it. Let's visualize the topics with the great [pyLDAvis](https://github.com/bmabey/pyLDAvis) tool. To do that, we first extract the relevant info from our model. We use BigARTM and it's not supported out of the box so we have a bit of work to do. If we'd have used Gensim or some other better-known (not better) library, this step would be a one-liner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bz2 import open as bz2_open\n",
    "from json import loads as json_loads\n",
    "from pickle import dump as pickle_dump, load as pickle_load\n",
    "from typing import Optional\n",
    "\n",
    "from numpy import ones as numpy_ones\n",
    "from pandas import DataFrame, read_csv as pandas_read_csv\n",
    "from pyLDAvis import prepare as pyldavis_prepare\n",
    "\n",
    "\n",
    "def prepare_visualization(artm_files_topics_path: str,\n",
    "                          artm_topics_identifiers_path: str,\n",
    "                          common_counter_path: str,\n",
    "                          filtered_identifiers_path: str,\n",
    "                          pyldavis_data_path: str):\n",
    "\n",
    "    def clean_artm_df(df: DataFrame,\n",
    "                      to_delete: str,\n",
    "                      transpose_name: Optional[str] = None):\n",
    "        del df[to_delete]\n",
    "        if transpose_name is not None:\n",
    "            df = df.T\n",
    "            df.index.name = transpose_name\n",
    "        return df\n",
    "\n",
    "    # We exchange rows and columns (transpose, .T) to have the topics as rows\n",
    "    # and the identifiers as columns\n",
    "    topics_identifiers_df = pandas_read_csv(\n",
    "        artm_topics_identifiers_path,\n",
    "        delimiter=\";\",\n",
    "        index_col=\"token\")\n",
    "    topics_identifiers_df = clean_artm_df(topics_identifiers_df, \"class_id\", \"topic\")\n",
    "    print(\"Start of the topics × identifiers dataframe:\")\n",
    "    display(topics_identifiers_df.head())\n",
    "\n",
    "    files_topics_df = pandas_read_csv(\n",
    "        artm_files_topics_path,\n",
    "        delimiter=\";\",\n",
    "        index_col=\"title\")\n",
    "    clean_artm_df(files_topics_df, \"id\")\n",
    "    print(\"Start of the files × topics dataframe:\")\n",
    "    display(files_topics_df.head())\n",
    "\n",
    "    files_topics_df /= files_topics_df.sum(axis=1)[:, None]\n",
    "    filler = (numpy_ones((files_topics_df.shape[1],))\n",
    "              / files_topics_df.shape[1])\n",
    "    for i, row in files_topics_df.iterrows():\n",
    "        if not (0.9 < row.sum() < 1.1):\n",
    "            files_topics_df.loc[i, :] = filler\n",
    "    vocab = topics_identifiers_df.columns\n",
    "    with bz2_open(filtered_identifiers_path, \"rt\", encoding=\"utf8\") as fh_rj, \\\n",
    "            open(common_counter_path, \"rb\") as fh_rp:\n",
    "        common_identifiers_counter = pickle_load(fh_rp)\n",
    "        doc_lengths_index = {}\n",
    "        for row_str in fh_rj:\n",
    "            row = json_loads(row_str)\n",
    "            doc_lengths_index[\n",
    "                \"%s//%s//%s\" % (\n",
    "                    row[\"repository_id\"],\n",
    "                    row[\"lang\"],\n",
    "                    row[\"file_path\"].replace(\" \", \"_\"))\n",
    "            ] = len(row[\"split_identifiers\"])\n",
    "        term_frequency = [common_identifiers_counter[t] for t in vocab]\n",
    "        doc_lengths = [doc_lengths_index[doc] for doc in files_topics_df.index]\n",
    "\n",
    "    with open(pyldavis_data_path, \"wb\") as fh:\n",
    "        pyldavis_data = pyldavis_prepare(topic_term_dists=topics_identifiers_df.values, \n",
    "                                         doc_topic_dists=files_topics_df.values,\n",
    "                                         doc_lengths=doc_lengths,\n",
    "                                         vocab=vocab,\n",
    "                                         term_frequency=term_frequency,\n",
    "                                         sort_topics=False)\n",
    "        pickle_dump(pyldavis_data, fh)\n",
    "\n",
    "\n",
    "prepare_visualization(run.path(Files.ARTM_FILES_TOPICS),\n",
    "                      run.path(Files.ARTM_TOPICS_IDENTIFIERS),\n",
    "                      run.path(Files.COMMON_IDENTIFIERS_COUNTER),\n",
    "                      run.path(Files.FILTERED_IDENTIFIERS),\n",
    "                      run.path(Files.PYLDAVIS_DATA))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now visualize the topics we just learned!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pickle import load as pickle_load\n",
    "\n",
    "from pyLDAvis import display as pyldavis_display\n",
    "\n",
    "\n",
    "def visualize(pyldavis_data_path: str):\n",
    "    with open(pyldavis_data_path, \"rb\") as fh:\n",
    "        pyldavis_data = pickle_load(fh)\n",
    "    return pyldavis_display(pyldavis_data)\n",
    "\n",
    "\n",
    "visualize(full_run.path(Files.PYLDAVIS_DATA))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Projects topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our learned topic model, we can now tackle our first task: understand what projects are about and find similar projects to existing ones based on their topics.\n",
    "\n",
    "To do that, we will compute the distance between the topics of all projects, and return the closest ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bz2 import open as bz2_open\n",
    "from json import loads as json_loads\n",
    "from pickle import dump as pickle_dump, load as pickle_load\n",
    "from re import compile as re_compile\n",
    "\n",
    "from numpy import sum as np_sum, vectorize\n",
    "from pandas import read_csv as pandas_read_csv\n",
    "\n",
    "\n",
    "def build_projects_topics(artm_files_topics_path: str,\n",
    "                          repos_topics_path: str):\n",
    "    files_topics_df = pandas_read_csv(artm_files_topics_path, delimiter=\";\")\n",
    "    all_but_repo_pattern = re_compile(r\"//.+//.+$\")\n",
    "    files_topics_df[\"repository_id\"] = files_topics_df[\"title\"].apply(\n",
    "        lambda x: all_but_repo_pattern.sub(\"\", x))\n",
    "    grouped_by_repo_df = files_topics_df.iloc[:, 2:].groupby(\"repository_id\")\n",
    "    repos_topics_df = grouped_by_repo_df.aggregate(np_sum)\n",
    "    repos_topics_df /= repos_topics_df.sum(axis=1)[:, None]\n",
    "    with open(repos_topics_path, \"wb\") as fh:\n",
    "        pickle_dump(repos_topics_df, fh)\n",
    "\n",
    "\n",
    "build_projects_topics(run.path(Files.ARTM_FILES_TOPICS),\n",
    "                      run.path(Files.REPOS_TOPICS))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Developers topics\n",
    "\n",
    "Now that we've computed file and project topics, let's compute topics for developers: we'll weight the topics of each file depending on how many lines each developers wrote in it. That'll give us a topic distribution for each developer!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bz2 import open as bz2_open\n",
    "from collections import Counter\n",
    "from json import loads as json_loads\n",
    "from multiprocessing import Pool\n",
    "from os.path import join as path_join\n",
    "from pickle import dump as pickle_dump\n",
    "from typing import Any, Dict\n",
    "\n",
    "from git import *\n",
    "\n",
    "\n",
    "def extract_author_stats(row: Dict[str, Any]):\n",
    "    row = json_loads(row)\n",
    "    repo = Repo(path_join(\"/devfest\", \"repos\", \"git-data\", row[\"repository_id\"]))\n",
    "    file_id = \"%s//%s//%s\" % (\n",
    "        row[\"repository_id\"],\n",
    "        row[\"lang\"],\n",
    "        row[\"file_path\"].replace(\" \", \"_\"))\n",
    "    commit_counter = Counter()\n",
    "    for blame_entry in repo.blame_incremental(\"HEAD\", row[\"file_path\"]):\n",
    "        commit_counter[blame_entry.commit] += blame_entry.linenos.stop - blame_entry.linenos.start\n",
    "    author_counter = Counter()\n",
    "    for commit, lines in commit_counter.items():\n",
    "        author = repo.git.show(\"-s\", \"--format=%ae\", str(commit))\n",
    "        author_counter[author] += lines\n",
    "    return file_id, author_counter\n",
    "\n",
    "\n",
    "def blame(filtered_identifiers_path: str,\n",
    "          contributions_path: str):\n",
    "    with bz2_open(filtered_identifiers_path, \"rt\", encoding=\"utf8\") as fh, \\\n",
    "            open(contributions_path, \"wb\") as fh_contributions, \\\n",
    "            Pool() as pool:\n",
    "        results = pool.map(extract_author_stats, fh.readlines())\n",
    "        pickle_dump(results, fh_contributions)\n",
    "\n",
    "\n",
    "blame(run.path(Files.FILTERED_IDENTIFIERS),\n",
    "      run.path(Files.CONTRIBUTIONS))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pickle import dump as pickle_dump, load as pickle_load\n",
    "\n",
    "from pandas import DataFrame, read_csv as pandas_read_csv\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "\n",
    "def build_authors_topics(contributions_path: str,\n",
    "                         artm_files_topics_path: str,\n",
    "                         authors_topics_path: str):\n",
    "    with open(contributions_path, \"rb\") as fh_contributions:\n",
    "        contribs = pickle_load(fh_contributions)\n",
    "    files_topics_df = pandas_read_csv(artm_files_topics_path, delimiter=\";\")\n",
    "    files_topics_df.set_index(\"title\", inplace=True)\n",
    "    del files_topics_df[\"id\"]\n",
    "    files_index = {f: i for i, f in enumerate(files_topics_df.index)}\n",
    "    authors = sorted(set().union(*(c for _, c in contribs)))\n",
    "    authors_index = {a: i for i, a in enumerate(authors)}\n",
    "    authors_topics_df = DataFrame(0,\n",
    "                                 index=authors,\n",
    "                                 columns=[\"topic_%d\" % i\n",
    "                                          for i in range(files_topics_df.shape[1])])\n",
    "    for file_id, counter in tqdm(contribs):\n",
    "        file_topics = files_topics_df.loc[file_id]\n",
    "        if len(file_topics.shape) > 1:\n",
    "            file_topics = file_topics.iloc[0, :].squeeze()\n",
    "        total = sum(counter.values())\n",
    "        for author, lines in counter.items():\n",
    "            authors_topics_df.loc[author, :] += file_topics * lines / total\n",
    "    authors_topics_df /= authors_topics_df.sum(axis=1)[:, None]\n",
    "    authors_topics_df.dropna(inplace=True)\n",
    "    with open(authors_topics_path, \"wb\") as fh:\n",
    "        pickle_dump(authors_topics_df, fh)\n",
    "\n",
    "\n",
    "build_authors_topics(run.path(Files.CONTRIBUTIONS),\n",
    "                     run.path(Files.ARTM_FILES_TOPICS),\n",
    "                     run.path(Files.AUTHORS_TOPICS))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Projects and developers search\n",
    "\n",
    "We have topics for projects and developers, great!\n",
    "\n",
    "Now we can compare them: we just have to define how far a given set of topics is from another and we're good to go. In the following cell we're using cosine similarity. It's widely used for that prupose and works quite well. Plus, `scikit-learn` has it already implemented if you don't want to write the (few) lines it requires :)\n",
    "\n",
    "With that defined, we can compare devs to devs, devs to projects, projects to devs and projects to projects! Let's go."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pickle import load as pickle_load\n",
    "\n",
    "from numpy import sum as vectorize\n",
    "from pandas import read_csv as pandas_read_csv\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "\n",
    "def build_comparison_functions(artm_topics_identifiers_path: str,\n",
    "                               repos_topics_path: str,\n",
    "                               authors_topics_path: str):\n",
    "    topics_identifiers_df = pandas_read_csv(artm_topics_identifiers_path, delimiter=\";\").T\n",
    "    topics_topk = topics_identifiers_df.iloc[1:, :].values.argsort()[:, -10:][:, ::-1]\n",
    "    vocab = topics_identifiers_df.iloc[0, :].values\n",
    "    topics_top_words = vectorize(lambda x: vocab[x])(topics_topk)\n",
    "    with open(authors_topics_path, \"rb\") as fh_authors_topics, \\\n",
    "            open(repos_topics_path, \"rb\") as fh_repos_topics:\n",
    "        authors_topics_df = pickle_load(fh_authors_topics)\n",
    "        repos_topics_df = pickle_load(fh_repos_topics)\n",
    "\n",
    "    for repo, topics_dist in repos_topics_df.iterrows():\n",
    "        topk = topics_dist.argsort()[-3:][::-1]\n",
    "        probk = topics_dist[topk]\n",
    "        # print(\"%s:\\n%s\" % (repo, \"\\n\".join(\"  %.2f: %s\" % (prob, \", \".join(topics_top_words[top + 1]))\n",
    "        #                                    for prob, top in zip(probk, topk))))\n",
    "\n",
    "    def build_comparison_function(df1, df2):\n",
    "        distances = cosine_similarity(df1.values, df2.values)\n",
    "        def f(key: str):\n",
    "            dist = distances[df1.index.get_loc(key)]\n",
    "            topk = dist.argsort()[-10:-1][::-1]\n",
    "            probk = dist[topk]\n",
    "            return [(df2.index[i], p) for i, p in zip(topk, probk)]\n",
    "        return f\n",
    "\n",
    "    return (build_comparison_function(repos_topics_df, repos_topics_df),\n",
    "            build_comparison_function(repos_topics_df, authors_topics_df),\n",
    "            build_comparison_function(authors_topics_df, repos_topics_df),\n",
    "            build_comparison_function(authors_topics_df, authors_topics_df))\n",
    "\n",
    "\n",
    "r_r, r_a, a_r, a_a = build_comparison_functions(\n",
    "    full_run.path(Files.ARTM_TOPICS_IDENTIFIERS),\n",
    "    full_run.path(Files.REPOS_TOPICS),\n",
    "    full_run.path(Files.AUTHORS_TOPICS)\n",
    ")\n",
    "\n",
    "\n",
    "def display_top(key, f):\n",
    "    top = f(key)\n",
    "    print(\"**********\")\n",
    "    print(\"Closest to %s\" % key)\n",
    "    for o, prob in top:\n",
    "        print(\"%60s: %.2f\" % (o, prob))\n",
    "    print(\"**********\")\n",
    "\n",
    "display_top(\"log4j\", r_r)\n",
    "display_top(\"log4j\", r_a)\n",
    "display_top(\"mwomack@apache.org\", a_a)\n",
    "display_top(\"lixiaojie_dev@outlook.com\", a_r)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}