Sematinc_Search_Spanish.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Sematinc_Search_Spanish.ipynb",
      "provenance": [],
      "machine_shape": "hm",
      "authorship_tag": "ABX9TyNgOQPXBnMxT4bt61T/uA88",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mrm8488/shared_colab_notebooks/blob/master/Sematinc_Search_Spanish.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iKWZPKJcoTt_",
        "colab_type": "text"
      },
      "source": [
        "# Finding similar documents with transformers (Spanish examples)\n",
        "\n",
        "## Finding signal in noise is hard, sometimes even for computers. Thankfully, transformers (a state of the art technique in NLP) can help us make sense of huge corpuses of documents.\n",
        "\n",
        "> Author: [Txus](https://www.codegram.com/blog/author/txus/)\n",
        "\n",
        "> Colab creator: [Manuel Romero](https://twitter.com/mrm8488)\n",
        "\n",
        "![Transformers](https://www.codegram.com/assets/static/finding-similar-documents-with-transformers.cover.a98966c.d7f8a36ba6ae230de36e74fac018f67b.jpg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aMdiWXeEpBpJ",
        "colab_type": "text"
      },
      "source": [
        "Whenever you have large amounts of text documents that you need to search, full-text search is a tried and true approach that has been popular for a long time. Specialized search engines such as ElasticSearch and even mainstream databases such as PostgreSQL support it.\n",
        "\n",
        "However, whenever language in those documents is highly context-dependent, full-text search falls apart. Human language is rich —words in different contexts have different meanings, and users can often find themselves sifting through meaningless results to find that one document they're looking for.\n",
        "\n",
        " ## Why not just stick with full-text search?\n",
        "Our goal is to be able to index a large number of documents and issue simple text queries similarly to a full-text search engine like ElasticSearch, but have them be context- and semantically aware.\n",
        "\n",
        "A big advantage for users of this search engine is that they won't have to sift through irrelevant documents just because of structural similarity (different documents containing similar words, even if used in different contexts).\n",
        "\n",
        "Perhaps more importantly, such a search engine has the capability to surface documents that are apparently different (they use different words altogether), but semantically very close —which would be clearly apparent to a human reading them, but not to a full-text search engine.\n",
        "\n",
        "## How can we understand context rather than just matching words?\n",
        "Recent advances in Natural Language Processing, namely the Transformer, have changed the scenery for good. Transformers are a family of neural networks that have seen great success in language modeling tasks (modeling statistical relationships between words in natural language).\n",
        "\n",
        "Recently, a type of transformer by Google Research called BERT has revolutionized the field of NLP. Since its advent, this type of neural network has become a bountiful area of research on its own.\n",
        "\n",
        "By using transformers to process documents, rather than just matching words like full-text search does, we can turn a very opaque representation (text) into compact, abstract representations, much easier to deal with programmatically.\n",
        "\n",
        "## Representing documents as vectors\n",
        "You can think of a transformer narrowly as a function from a piece of text to a vector, or array of numbers. Importantly, we want this vector to always have a constant length, so that we can easily compare it to other vectors extracted from other documents.\n",
        "\n",
        "A bit more philosophically, you can think of each number in the vector as a coordinate in an N-dimensional space (where N is the length of the vector). The working assumption here is that, if the transformer learned a useful representation of the document, similar documents will live close together in that N-dimensional space.\n",
        "\n",
        " ## Finding similar documents\n",
        "So, now we have a bunch of vectors extracted from our documents. If they depict an N-dimensional space where similar documents live close to each other, we just need to find the nearest documents in that space, that is, where the coordinates for each dimension are closest to each other.\n",
        "\n",
        "**ElasticSearch** happens to support a dense_vector type, which can index such vectors, and ranking results according to the Euclidean distance to a specific point in that N-dimensional space. Very convenient, however, for the purposes of this demonstration, we'll use an in-memory database that can index dense vectors as well.\n",
        "\n",
        "Let's try it out!\n",
        "How about we try to hack a prototype in 5 minutes? We can use HuggingFace's transformers library for the highest convenience, and as mentioned, instead of ElasticSearch we'll use an in-memory vector search library called faiss."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gekw_WOPpc6v",
        "colab_type": "text"
      },
      "source": [
        "Let's dive into code!!! "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eS6LVkSUhHyz",
        "colab_type": "text"
      },
      "source": [
        "Install required libaries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NXYFvwJ-gZBE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 680
        },
        "outputId": "1c03c016-e3d6-4505-f492-018f656cbff7"
      },
      "source": [
        "!pip install -q transformers"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting transformers\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/27/3c/91ed8f5c4e7ef3227b4119200fc0ed4b4fd965b1f0172021c25701087825/transformers-3.0.2-py3-none-any.whl (769kB)\n",
            "\u001b[K     |████████████████████████████████| 778kB 9.7MB/s \n",
            "\u001b[?25hCollecting faiss\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/bd/1c/4ae6cb87cf0c09c25561ea48db11e25713b25c580909902a92c090b377c0/faiss-1.5.3-cp36-cp36m-manylinux1_x86_64.whl (4.7MB)\n",
            "\u001b[K     |████████████████████████████████| 4.7MB 33.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.6.0+cu101)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers) (1.18.5)\n",
            "Collecting sacremoses\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz (883kB)\n",
            "\u001b[K     |████████████████████████████████| 890kB 70.6MB/s \n",
            "\u001b[?25hRequirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from transformers) (0.7)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers) (2019.12.20)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers) (4.41.1)\n",
            "Collecting tokenizers==0.8.1.rc1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/40/d0/30d5f8d221a0ed981a186c8eb986ce1c94e3a6e87f994eae9f4aa5250217/tokenizers-0.8.1rc1-cp36-cp36m-manylinux1_x86_64.whl (3.0MB)\n",
            "\u001b[K     |████████████████████████████████| 3.0MB 62.3MB/s \n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers) (3.0.12)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from transformers) (20.4)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers) (2.23.0)\n",
            "Collecting sentencepiece!=0.1.92\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d4/a4/d0a884c4300004a78cca907a6ff9a5e9fe4f090f5d95ab341c53d28cbc58/sentencepiece-0.1.91-cp36-cp36m-manylinux1_x86_64.whl (1.1MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1MB 53.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch) (0.16.0)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (1.15.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (0.16.0)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers) (2.4.7)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2020.6.20)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2.10)\n",
            "Building wheels for collected packages: sacremoses\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893260 sha256=70d7cfb1976d03cd200f9a133e229a36a80adee9a78818dd2bf1ed9c86f60d6c\n",
            "  Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45\n",
            "Successfully built sacremoses\n",
            "Installing collected packages: sacremoses, tokenizers, sentencepiece, transformers, faiss\n",
            "Successfully installed faiss-1.5.3 sacremoses-0.0.43 sentencepiece-0.1.91 tokenizers-0.8.1rc1 transformers-3.0.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Yl3V9cu-j80P",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "46cffe26-27cc-4681-c029-44ec858b34d4"
      },
      "source": [
        "!pip install faiss-gpu"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting faiss-gpu\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a8/69/0e3f56024bb1423a518287673071ae512f9965d1faa6150deef5cc9e7996/faiss_gpu-1.6.3-cp36-cp36m-manylinux2010_x86_64.whl (35.5MB)\n",
            "\u001b[K     |████████████████████████████████| 35.5MB 88kB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from faiss-gpu) (1.18.5)\n",
            "Installing collected packages: faiss-gpu\n",
            "Successfully installed faiss-gpu-1.6.3\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FGBgfPCGg61-",
        "colab_type": "text"
      },
      "source": [
        "For our purposes we'll use ```RuPERTa-base``` model in Spanish, but there are tons of other models available in the transformers library, also for other languages and use cases. You can also try ```electricidad-small-discriminator``` for Spanish"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X4kng4taiCjR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from transformers import AutoModel, AutoTokenizer\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"mrm8488/RuPERTa-base\")\n",
        "model = AutoModel.from_pretrained(\"mrm8488/RuPERTa-base\")"
      ],
      "execution_count": 56,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kXuT2-bhhh6x",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "documents = [\n",
        "    \"El restaurante no era tan bueno como la peícula que vimos.\",\n",
        "    \"Vendo coche usado en buenas condiciones\",\n",
        "    \"La comida estaba bien, pero el resto regular\",\n",
        "    \"Me gustan los gatos, pero odio las hienas\",\n",
        "    \"En la carretera conduce con precaución\",\n",
        "]"
      ],
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4PbKcw3ygxFb",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "6a9f7eef-70a3-4385-e6a3-ecd3736054dd"
      },
      "source": [
        "vectors = [\n",
        "  # tokenize the document, return it as PyTorch tensors (vectors),\n",
        "  # and pass it onto the model\n",
        "  model(**tokenizer(document, return_tensors='pt'))[0].detach().squeeze()\n",
        "  for document in documents\n",
        "]\n",
        "\n",
        "[v.size() for v in vectors]"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[torch.Size([15, 768]),\n",
              " torch.Size([9, 768]),\n",
              " torch.Size([11, 768]),\n",
              " torch.Size([13, 768]),\n",
              " torch.Size([8, 768])]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D4xC7g63iO4t",
        "colab_type": "text"
      },
      "source": [
        "We encoded the documents into vectors, but we see a little issue —these encodings are vectors of size 15x768, 12x768, 10x768, 15x768 and 10x768 respectively. Since they have different sizes, they cannot be compared. Why is that?\n",
        "\n",
        "These vectors are not yet the final representation we want for our documents —they are 768-dimensional vectors for each token in the document. Since documents have a different number of tokens (because some texts are longer than others), we end up with this.\n",
        "\n",
        "Rather than the coordinates of each token in the 768-dimensional space, we want to find the general coordinates of the document, which we can do by averaging the points and finding the center. (There are more sophisticated ways, but this will do for now.)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iW_PQHhjiYQl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "f2727b22-0ec6-40b0-836e-3de85433ea45"
      },
      "source": [
        "import torch\n",
        "\n",
        "averaged_vectors = [torch.mean(vector, dim=0) for vector in vectors]\n",
        "\n",
        "[v.size() for v in averaged_vectors]"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[torch.Size([768]),\n",
              " torch.Size([768]),\n",
              " torch.Size([768]),\n",
              " torch.Size([768]),\n",
              " torch.Size([768])]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OoGt3doUiqgR",
        "colab_type": "text"
      },
      "source": [
        "We got it! A unified, compact representation for each of our documents, all of size 768. Let's pack it in a function we can reuse:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p7qgir9biidJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def encode(document: str) -> torch.Tensor:\n",
        "  tokens = tokenizer(document, return_tensors='pt')\n",
        "  vector = model(**tokens)[0].detach().squeeze()\n",
        "  return torch.mean(vector, dim=0)"
      ],
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-71iLuqJi2XY",
        "colab_type": "text"
      },
      "source": [
        "Now let's index those documents in an in-memory vector-space database, to test:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rCf8ahhRixeI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import faiss\n",
        "import numpy as np\n",
        "\n",
        "index = faiss.IndexIDMap(faiss.IndexFlatIP(768)) # the size of our vector space\n",
        "# index all the documents, we need them as numpy arrays first\n",
        "index.add_with_ids(\n",
        "    np.array([t.numpy() for t in averaged_vectors]),\n",
        "    # the IDs will be 0 to len(documents)\n",
        "    np.array(range(0, len(documents)))\n",
        ")"
      ],
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-soV3u6fjbbM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def search(query: str, k=1):\n",
        "  encoded_query = encode(query).unsqueeze(dim=0).numpy()\n",
        "  top_k = index.search(encoded_query, k)\n",
        "  scores = top_k[0][0]\n",
        "  results = [documents[_id] for _id in top_k[1][0]]\n",
        "  return list(zip(results, scores))"
      ],
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7la6-LKjnzB",
        "colab_type": "text"
      },
      "source": [
        "Now we can try to search documents similar to a specific one:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8IsYj-p_jgdY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "638dbdfc-3387-4645-f7ec-1d7dd5c067c4"
      },
      "source": [
        "documents[1]"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Vendo coche usado en buenas condiciones'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pxrTlKOVjxze",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "ca77805d-9979-4a50-9687-bd62af1f5eaa"
      },
      "source": [
        "search(documents[1], k=2)"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Vendo coche usado en buenas condiciones', 233.06683),\n",
              " ('En la carretera conduce con precaución', 198.67575)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S7nAe6XAkyQ-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "61f0f0cf-bc21-4da5-856c-285fc4d5b7a5"
      },
      "source": [
        "search(documents[0], k=2)"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('El restaurante no era tan bueno como la peícula que vimos.', 223.23514),\n",
              " ('La comida estaba bien, pero el resto regular', 207.92018)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QBt2V6_Rk-cR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "00d898f8-23fb-43ef-af70-634d3ae535bb"
      },
      "source": [
        "search(\"Sé conducir\", k=2)"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Vendo coche usado en buenas condiciones', 197.57762),\n",
              " ('En la carretera conduce con precaución', 192.79391)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UIC4r7l1lNrz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "f903f47e-2e5d-4f7e-899c-cf4af74a8407"
      },
      "source": [
        "search(\"El plan era ver una película e ir a cenar\", k=2)"
      ],
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('La comida estaba bien, pero el resto regular', 191.00691),\n",
              " ('El restaurante no era tan bueno como la peícula que vimos.', 185.60252)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 63
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0f0VWYCZlWJJ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "e4f99f8c-7547-4ed3-b848-b54a4e73d2ec"
      },
      "source": [
        "search(\"Me dan miedo los animales salvajes\", k=2)"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Me gustan los gatos, pero odio las hienas', 182.2342),\n",
              " ('El restaurante no era tan bueno como la peícula que vimos.', 177.5228)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fl267ulqrOCa",
        "colab_type": "text"
      },
      "source": [
        "## Conclusion\n",
        "In this blogpost we've learned how transformers, the current state of the art in Natural Language Processing, can help us distill text documents into points in N-dimensional vector spaces.\n",
        "\n",
        "By searching by distance to points in that space, we can discover documents similar to each other, as well as similar to user-crafted queries, creating a semantic search engine in a few lines of Python.\n",
        "\n",
        "As a next step, I encourage you to try different models and languages (you can find them all in HuggingFace's Model Hub) and also, try using ElasticSearch dense_vectors to index your documents there and take advantage of multi-faceted, production-ready search in vector space."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D8XRbRIMrm2P",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}