demo/cand_generator_nmslib.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using k-NN search on dense and dense-sparse representation for candidate generation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Two things to do before we start:\n",
    "1. Point environment variable `COLLECT_ROOT` to the collection root.\n",
    "2. Change directory to the location of installed scripts/binaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: COLLECT_ROOT=/home/leo/flexneuart_collections\n"
     ]
    }
   ],
   "source": [
    "%env COLLECT_ROOT=/home/leo/flexneuart_collections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/leo/flexneuart_scripts\n"
     ]
    }
   ],
   "source": [
    "cd /home/leo/flexneuart_scripts/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating purely dense (ANCE) embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exporting data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p $COLLECT_ROOT/wikipedia_dpr_nq_sample/derived_data/nmslib/ance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using collection root: /home/leo/flexneuart_collections\n",
      "==========================================================================\n",
      "Collection directory:      /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample                                    \n",
      "Extractor JSON:            exper_desc.best/extractors/ance.json                                      \n",
      "Output file:               /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/derived_data/nmslib/ance/export.data                                       \n",
      "Forward index directory:   forward_index                                   \n",
      "Model 1 directory:         derived_data/giza                                  \n",
      "Embedding directory:       derived_data/embeddings                                   \n",
      "==========================================================================\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.resources.ResourceManager - Resource manager initialization. Resource root:/home/leo/flexneuart_collections/wikipedia_dpr_nq_sample\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.fwdindx.MapDbBackend - MapDB opened for reading: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/forward_index/dense.mapdb_dataDict\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.fwdindx.ForwardIndexBinaryDataDict - Finished loading context from file: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/forward_index/dense.mapdb_dataDict\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.letor.FeatExtrDenseDocEmbedDotProdSimilarity - Index field name: dense normalize embeddings:? false\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Writing the number of entries (774392) to the output file\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 100000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 200000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 300000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 400000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 500000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 600000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 700000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 774392 docs\n"
     ]
    }
   ],
   "source": [
    "!./export_nmslib/export_nmslib_dense_sparse_fused.sh \\\n",
    "    wikipedia_dpr_nq_sample \\\n",
    "    exper_desc.best/extractors/ance.json \\\n",
    "    nmslib/ance/export.data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to compile the latest NMSLIB [query server (details)](https://github.com/nmslib/nmslib/blob/master/manual/query_server.md). Then it needs to be started from the FlexNeuART source tree root as shown below. In our example, the server is going to carry out the brute-force search. It is also possible to create an index using, e.g., HNSW:\n",
    "\n",
    "```\n",
    "export COLLECT_ROOT=/home/leo/flexneuart_collections\n",
    "<path to the query server>/query_server \\\n",
    "    -p 8000 \\\n",
    "     -s sparse_dense_fusion:weightfilename=$COLLECT_ROOT/wikipedia_dpr_nq_sample/exper_desc.best/nmslib/ance/fusion_weights \\\n",
    "     -i $COLLECT_ROOT/wikipedia_dpr_nq_sample/derived_data/nmslib/ance/export.data \\\n",
    "     -m brute_force\n",
    "\n",
    "\n",
    "```\n",
    "\n",
    "Now we can run a benchmark:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using collection root: /home/leo/flexneuart_collections\n",
      "The number of CPU cores:      8\n",
      "The number of || experiments: 1\n",
      "The number of threads:        8\n",
      "================================================================================\n",
      "Experiment descriptor file:                                 /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/exper_desc.best/nmslib_ance.json\n",
      "Default test set:                                           dev\n",
      "Number of parallel experiments:                             1\n",
      "Number of threads in feature extractors/query applications: 8\n",
      "================================================================================\n",
      "Parsed experiment parameters:\n",
      "experSubdir:final_exper/ance_knn\n",
      "candProv:nmslib\n",
      "candProvAddConf:exper_desc.best/nmslib/ance/cand_prov.json\n",
      "candProvURI:localhost:8000\n",
      "candQty:1000\n",
      "testOnly:1\n",
      "========================================\n",
      "Experimental directory already exists (removing contents): /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/ance_knn\n",
      "Cleaning the experimental directory: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/ance_knn\n",
      "Started a process 15324, working dir: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/ance_knn\n",
      "Process log file: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/ance_knn/exper.log\n",
      "Waiting for 1 child processes\n",
      "Process with pid=15324 finished successfully.\n",
      "Waiting for 0 child processes\n",
      "================================================================================\n",
      "1 experiments executed\n",
      "0 experiments failed\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "!./exper/run_experiments.sh \\\n",
    "    wikipedia_dpr_nq_sample \\\n",
    "    exper_desc.best/nmslib_ance.json \\\n",
    "    -test_part dev -clean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Top-100 report:__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of queries:    2500\r\n",
      "NDCG@10:  0.705800\r\n",
      "NDCG@20:  0.702400\r\n",
      "NDCG@100: 0.728700\r\n",
      "P20:      0.145400\r\n",
      "MAP:      0.595500\r\n",
      "MRR:      0.946100\r\n",
      "Recall:   0.546620\r\n"
     ]
    }
   ],
   "source": [
    "!cat $COLLECT_ROOT/wikipedia_dpr_nq_sample/results/dev/final_exper/ance_knn/rep/out_100.rep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating mixed dense-sparse (BM25+ANCE) embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exporting data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p $COLLECT_ROOT/wikipedia_dpr_nq_sample/derived_data/nmslib/bm25_ance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using collection root: /home/leo/flexneuart_collections\n",
      "==========================================================================\n",
      "Collection directory:      /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample                                    \n",
      "Extractor JSON:            exper_desc.best/extractors/bm25_ance.json                                      \n",
      "Output file:               /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/derived_data/nmslib/bm25_ance/export.data                                       \n",
      "Forward index directory:   forward_index                                   \n",
      "Model 1 directory:         derived_data/giza                                  \n",
      "Embedding directory:       derived_data/embeddings                                   \n",
      "==========================================================================\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.resources.ResourceManager - Resource manager initialization. Resource root:/home/leo/flexneuart_collections/wikipedia_dpr_nq_sample\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.fwdindx.MapDbBackend - MapDB opened for reading: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/forward_index/text.mapdb_dataDict\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.fwdindx.ForwardIndexBinaryDataDict - Finished loading context from file: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/forward_index/text.mapdb_dataDict\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.fwdindx.MapDbBackend - MapDB opened for reading: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/forward_index/dense.mapdb_dataDict\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.fwdindx.ForwardIndexBinaryDataDict - Finished loading context from file: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/forward_index/dense.mapdb_dataDict\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.letor.FeatExtrDenseDocEmbedDotProdSimilarity - Index field name: dense normalize embeddings:? false\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Writing the number of entries (774392) to the output file\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 100000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 200000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 300000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 400000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 500000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 600000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 700000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 774392 docs\n"
     ]
    }
   ],
   "source": [
    "!./export_nmslib/export_nmslib_dense_sparse_fused.sh \\\n",
    "    wikipedia_dpr_nq_sample \\\n",
    "    exper_desc.best/extractors/bm25_ance.json \\\n",
    "    nmslib/bm25_ance/export.data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to compile the latest NMSLIB [query server (details)](https://github.com/nmslib/nmslib/blob/master/manual/query_server.md). Then it needs to be started from the FlexNeuART source tree root as shown below. In our example, the server is going to carry out the brute-force search. It is also possible to create an index using, e.g., HNSW:\n",
    "\n",
    "```\n",
    "<path to the query server>/query_server \\\n",
    "    -p 8000 \\\n",
    "     -s sparse_dense_fusion:weightfilename=\\\n",
    "$COLLECT_ROOT/wikipedia_dpr_nq_sample/exper_desc.best/nmslib/bm25_ance/fusion_weights \\\n",
    "     -i $COLLECT_ROOT/wikipedia_dpr_nq_sample/derived_data/nmslib/bm25_ance/export.data \\\n",
    "     -m brute_force\n",
    "```\n",
    "\n",
    "Now we can run a benchmark:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using collection root: /home/leo/flexneuart_collections\n",
      "The number of CPU cores:      8\n",
      "The number of || experiments: 1\n",
      "The number of threads:        8\n",
      "================================================================================\n",
      "Experiment descriptor file:                                 /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/exper_desc.best/nmslib_bm25_ance.json\n",
      "Default test set:                                           dev\n",
      "Number of parallel experiments:                             1\n",
      "Number of threads in feature extractors/query applications: 8\n",
      "================================================================================\n",
      "Parsed experiment parameters:\n",
      "experSubdir:final_exper/bm25_ance_knn\n",
      "candProv:nmslib\n",
      "candProvAddConf:exper_desc.best/nmslib/bm25_ance/cand_prov.json\n",
      "candProvURI:localhost:8000\n",
      "candQty:1000\n",
      "testOnly:1\n",
      "========================================\n",
      "Started a process 15552, working dir: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25_ance_knn\n",
      "Process log file: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25_ance_knn/exper.log\n",
      "Waiting for 1 child processes\n",
      "Process with pid=15552 finished successfully.\n",
      "Waiting for 0 child processes\n",
      "================================================================================\n",
      "1 experiments executed\n",
      "0 experiments failed\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "!./exper/run_experiments.sh \\\n",
    "    wikipedia_dpr_nq_sample \\\n",
    "    exper_desc.best/nmslib_bm25_ance.json \\\n",
    "    -test_part dev"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Top 100 report__:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of queries:    2500\r\n",
      "NDCG@10:  0.714900\r\n",
      "NDCG@20:  0.712100\r\n",
      "NDCG@100: 0.740300\r\n",
      "P20:      0.151700\r\n",
      "MAP:      0.606000\r\n",
      "MRR:      0.945300\r\n",
      "Recall:   0.577034\r\n"
     ]
    }
   ],
   "source": [
    "!cat $COLLECT_ROOT/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25_ance_knn/rep/out_100.rep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating mixed dense-sparse (BM25+ANCE) embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Note, we expect this mode is going primarily useful for purely sparse mixes. For mostly dense mixes, the efficiency is subpar compared to dense-sparse export"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exporting data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p $COLLECT_ROOT/wikipedia_dpr_nq_sample/derived_data/nmslib/bm25_ance_interleaved"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using collection root: /home/leo/flexneuart_collections\n",
      "==========================================================================\n",
      "Collection directory:      /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample                                    \n",
      "Extractor JSON:            exper_desc.best/extractors/bm25_ance.json                                      \n",
      "Output file:               /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/derived_data/nmslib/bm25_ance_interleaved/export.data                                       \n",
      "Forward index directory:   forward_index                                   \n",
      "Model 1 directory:         derived_data/giza                                  \n",
      "Embedding directory:       derived_data/embeddings                                   \n",
      "Model file:                exper_desc.best/models/bm25_ance.model                                     \n",
      "==========================================================================\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.resources.ResourceManager - Resource manager initialization. Resource root:/home/leo/flexneuart_collections/wikipedia_dpr_nq_sample\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.fwdindx.MapDbBackend - MapDB opened for reading: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/forward_index/text.mapdb_dataDict\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.fwdindx.ForwardIndexBinaryDataDict - Finished loading context from file: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/forward_index/text.mapdb_dataDict\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.fwdindx.MapDbBackend - MapDB opened for reading: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/forward_index/dense.mapdb_dataDict\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.fwdindx.ForwardIndexBinaryDataDict - Finished loading context from file: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/forward_index/dense.mapdb_dataDict\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.letor.FeatExtrDenseDocEmbedDotProdSimilarity - Index field name: dense normalize embeddings:? false\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBSparse - Writing the number of entries (774392) to the output file\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBSparse - Exported 100000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBSparse - Exported 200000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBSparse - Exported 300000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBSparse - Exported 400000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBSparse - Exported 500000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBSparse - Exported 600000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBSparse - Exported 700000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBSparse - Exported 774392 docs\n"
     ]
    }
   ],
   "source": [
    "!./export_nmslib/export_nmslib_sparse.sh \\\n",
    "    wikipedia_dpr_nq_sample \\\n",
    "    exper_desc.best/extractors/bm25_ance.json \\\n",
    "    nmslib/bm25_ance_interleaved/export.data  \\\n",
    "    -model_file exper_desc.best/models/bm25_ance.model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to compile the latest NMSLIB [query server (details)](https://github.com/nmslib/nmslib/blob/master/manual/query_server.md). Then it needs to be started from the FlexNeuART source tree root as shown below. In our example, the server is going to carry out the brute-force search. It is also possible to create an index using, e.g., HNSW. The __difference__ from the previous examples is that the similarity function here is a simple unweighted inner product between sparse vectors:\n",
    "\n",
    "```\n",
    "<path to the query server>/query_server \\\n",
    "    -p 8000 \\\n",
    "     -s negdotprod_sparse_bin_fast \\\n",
    "     -i $COLLECT_ROOT/wikipedia_dpr_nq_sample/derived_data/nmslib/bm25_ance_interleaved/export.data \\\n",
    "     -m brute_force\n",
    "```\n",
    "\n",
    "Now we can run a benchmark:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using collection root: /home/leo/flexneuart_collections\n",
      "The number of CPU cores:      8\n",
      "The number of || experiments: 1\n",
      "The number of threads:        4\n",
      "================================================================================\n",
      "Experiment descriptor file:                                 /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/exper_desc.best/nmslib_bm25_ance_interleaved.json\n",
      "Default test set:                                           dev\n",
      "Number of parallel experiments:                             1\n",
      "Number of threads in feature extractors/query applications: 4\n",
      "================================================================================\n",
      "Parsed experiment parameters:\n",
      "experSubdir:final_exper/bm25_ance_knn_interleaved\n",
      "candProv:nmslib\n",
      "candProvAddConf:exper_desc.best/nmslib/bm25_ance_interleaved/cand_prov.json\n",
      "candProvURI:localhost:8000\n",
      "candQty:1000\n",
      "testOnly:1\n",
      "========================================\n",
      "Experimental directory already exists (removing contents): /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25_ance_knn_interleaved\n",
      "Cleaning the experimental directory: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25_ance_knn_interleaved\n",
      "Started a process 15898, working dir: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25_ance_knn_interleaved\n",
      "Process log file: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25_ance_knn_interleaved/exper.log\n",
      "Waiting for 1 child processes\n",
      "Process with pid=15898 finished successfully.\n",
      "Waiting for 0 child processes\n",
      "================================================================================\n",
      "1 experiments executed\n",
      "0 experiments failed\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "!./exper/run_experiments.sh \\\n",
    "    wikipedia_dpr_nq_sample \\\n",
    "    exper_desc.best/nmslib_bm25_ance_interleaved.json \\\n",
    "    -test_part dev \\\n",
    "    -thread_qty 4 \\\n",
    "    -clean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Top 100 report__:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of queries:    2500\r\n",
      "NDCG@10:  0.714900\r\n",
      "NDCG@20:  0.712100\r\n",
      "NDCG@100: 0.740300\r\n",
      "P20:      0.151700\r\n",
      "MAP:      0.606000\r\n",
      "MRR:      0.945300\r\n",
      "Recall:   0.577034\r\n"
     ]
    }
   ],
   "source": [
    "!cat $COLLECT_ROOT/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25_ance_knn/rep/out_100.rep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating mixed dense-sparse (BM25+glove averaged) embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exporting data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p $COLLECT_ROOT/wikipedia_dpr_nq_sample/derived_data/nmslib/bm25_avgembed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using collection root: /home/leo/flexneuart_collections\n",
      "==========================================================================\n",
      "Collection directory:      /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample                                    \n",
      "Extractor JSON:            exper_desc.best/extractors/bm25_avgembed.json                                      \n",
      "Output file:               /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/derived_data/nmslib/bm25_avgembed/export.data                                       \n",
      "Forward index directory:   forward_index                                   \n",
      "Model 1 directory:         derived_data/giza                                  \n",
      "Embedding directory:       derived_data/embeddings                                   \n",
      "Model file:                                                     \n",
      "==========================================================================\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.resources.ResourceManager - Resource manager initialization. Resource root:/home/leo/flexneuart_collections/wikipedia_dpr_nq_sample\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.fwdindx.MapDbBackend - MapDB opened for reading: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/forward_index/text.mapdb_dataDict\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.fwdindx.ForwardIndexBinaryDataDict - Finished loading context from file: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/forward_index/text.mapdb_dataDict\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.fwdindx.MapDbBackend - MapDB opened for reading: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/forward_index/text_unlemm.mapdb_dataDict\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.fwdindx.ForwardIndexBinaryDataDict - Finished loading context from file: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/forward_index/text_unlemm.mapdb_dataDict\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.letor.EmbeddingReaderAndRecoder - Loaded 50000 source word vectors from '/home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/derived_data/embeddings/glove/glove.6B.50d.txt.bz2'\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.letor.EmbeddingReaderAndRecoder - Loaded 100000 source word vectors from '/home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/derived_data/embeddings/glove/glove.6B.50d.txt.bz2'\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.letor.EmbeddingReaderAndRecoder - Loaded 150000 source word vectors from '/home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/derived_data/embeddings/glove/glove.6B.50d.txt.bz2'\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.letor.EmbeddingReaderAndRecoder - Loaded 200000 source word vectors from '/home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/derived_data/embeddings/glove/glove.6B.50d.txt.bz2'\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.letor.EmbeddingReaderAndRecoder - Loaded 250000 source word vectors from '/home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/derived_data/embeddings/glove/glove.6B.50d.txt.bz2'\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.letor.EmbeddingReaderAndRecoder - Loaded 300000 source word vectors from '/home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/derived_data/embeddings/glove/glove.6B.50d.txt.bz2'\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.letor.EmbeddingReaderAndRecoder - Loaded 350000 source word vectors from '/home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/derived_data/embeddings/glove/glove.6B.50d.txt.bz2'\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.letor.EmbeddingReaderAndRecoder - Loaded 400000 source word vectors from '/home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/derived_data/embeddings/glove/glove.6B.50d.txt.bz2'\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.letor.EmbeddingReaderAndRecoder - Finished loading 271158 word vectors from '/home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/derived_data/embeddings/glove/glove.6B.50d.txt.bz2' (out of 400000), dimensionality: 50\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Writing the number of entries (774392) to the output file\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 100000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 200000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 300000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 400000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 500000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 600000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 700000 docs\n",
      "[main] INFO edu.cmu.lti.oaqa.flexneuart.apps.ExportToNMSLIBDenseSparseFusion - Exported 774392 docs\n"
     ]
    }
   ],
   "source": [
    "!./export_nmslib/export_nmslib_dense_sparse_fused.sh \\\n",
    "    wikipedia_dpr_nq_sample \\\n",
    "    exper_desc.best/extractors/bm25_avgembed.json \\\n",
    "    nmslib/bm25_avgembed/export.data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to compile the latest NMSLIB [query server (details)](https://github.com/nmslib/nmslib/blob/master/manual/query_server.md). Then it needs to be started from the FlexNeuART source tree root as shown below. In our example, the server is going to carry out the brute-force search. It is also possible to create an index using, e.g., HNSW. The __difference__ from the previous examples is that the similarity function here is a simple unweighted inner product between sparse vectors:\n",
    "\n",
    "```\n",
    "<path to the query server>/query_server \\\n",
    "    -p 8000 \\\n",
    "     -s sparse_dense_fusion:weightfilename=\\\n",
    "$COLLECT_ROOT/wikipedia_dpr_nq_sample/exper_desc.best/nmslib/bm25_avgembed/fusion_weights \\\n",
    "     -i $COLLECT_ROOT/wikipedia_dpr_nq_sample/derived_data/nmslib/bm25_avgembed/export.data \\\n",
    "     -m brute_force\n",
    "```\n",
    "\n",
    "Now we can run a benchmark:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using collection root: /home/leo/flexneuart_collections\n",
      "The number of CPU cores:      8\n",
      "The number of || experiments: 1\n",
      "The number of threads:        4\n",
      "================================================================================\n",
      "Experiment descriptor file:                                 /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/exper_desc.best/nmslib_bm25_avgembed.json\n",
      "Default test set:                                           dev\n",
      "Number of parallel experiments:                             1\n",
      "Number of threads in feature extractors/query applications: 4\n",
      "================================================================================\n",
      "Parsed experiment parameters:\n",
      "experSubdir:final_exper/bm25_avgembed_knn\n",
      "candProv:nmslib\n",
      "candProvAddConf:exper_desc.best/nmslib/bm25_avgembed/cand_prov.json\n",
      "candProvURI:localhost:8000\n",
      "candQty:1000\n",
      "testOnly:1\n",
      "========================================\n",
      "Started a process 16166, working dir: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25_avgembed_knn\n",
      "Process log file: /home/leo/flexneuart_collections/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25_avgembed_knn/exper.log\n",
      "Waiting for 1 child processes\n",
      "Process with pid=16166 finished successfully.\n",
      "Waiting for 0 child processes\n",
      "================================================================================\n",
      "1 experiments executed\n",
      "0 experiments failed\n",
      "================================================================================\n"
     ]
    }
   ],
   "source": [
    "!./exper/run_experiments.sh \\\n",
    "    wikipedia_dpr_nq_sample \\\n",
    "    exper_desc.best/nmslib_bm25_avgembed.json \\\n",
    "    -test_part dev \\\n",
    "    -thread_qty 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Top 100 report__:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of queries:    2500\r\n",
      "NDCG@10:  0.402300\r\n",
      "NDCG@20:  0.434900\r\n",
      "NDCG@100: 0.507800\r\n",
      "P20:      0.165600\r\n",
      "MAP:      0.345600\r\n",
      "MRR:      0.486600\r\n",
      "Recall:   0.820033\r\n"
     ]
    }
   ],
   "source": [
    "!cat $COLLECT_ROOT/wikipedia_dpr_nq_sample/results/dev/final_exper/bm25_avgembed_knn/rep/out_100.rep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}