docs/it-threat-detection.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "PFHpj5RBRAea"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/docs/it-threat-detection.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/docs/it-threat-detection.ipynb)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "b4Yv1jeGywpL"
      },
      "source": [
        "## IT Threat Detection With Similarity Search\n",
        "\n",
        "[![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/full-link.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/security/it-threat-detection.ipynb)\n",
        "\n",
        "This notebook shows how to use Pinecone's similarity search as a service to build an application for detecting rare events. Such application is common in cyber-security and fraud detection domains wherein only a tiny fraction of the events are malicious.\n",
        "\n",
        "Here we will build a network intrusion detector. Network intrusion detection systems monitor incoming and outgoing network traffic flow, raising alarms whenever a threat is detected. Here we use a deep-learning model and similarity search in detecting and classifying network intrusion traffic.\n",
        "\n",
        "We will start by indexing a set of labeled traffic events in the form of vector embeddings. Each event is either benign or malicious. The vector embeddings are rich, mathematical representations of the network traffic events. It is making it possible to determine how similar the network events are to one another using similarity-search algorithms built into Pinecone. Here we will transform network traffic events into vectors using a deep learning model from recent academic work.\n",
        "\n",
        "\n",
        "We will then take some new (unseen) network events and search through the index to find the most similar matches, along with their labels. In such a way, we will propagate the matched labels to classify the unseen events as benign or malicious. Mind that the intrusion detection task is a challenging classification task because malicious events are sporadic. The similarity search service helps us sift the most relevant historical labeled events. That way, we identify these rare events while keeping a low rate of false alarms.\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "1OUSClcPhU4j"
      },
      "source": [
        "## Setting up Pinecone\n",
        "\n",
        "We will first install and initialize Pinecone. You can get your [API Key here](https://app.pinecone.io)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "251n1avKzCrm",
        "outputId": "77b87d58-67dc-45ac-e242-e1e1fd36aa43"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m179.1/179.1 kB\u001b[0m \u001b[31m2.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m60.0/60.0 kB\u001b[0m \u001b[31m4.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m300.0/300.0 kB\u001b[0m \u001b[31m14.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m29.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m23.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m12.3/12.3 MB\u001b[0m \u001b[31m39.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m16.4/16.4 MB\u001b[0m \u001b[31m36.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m38.9/38.9 MB\u001b[0m \u001b[31m8.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m74.5/74.5 kB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m341.8/341.8 kB\u001b[0m \u001b[31m19.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m72.9/72.9 kB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m10.9/10.9 MB\u001b[0m \u001b[31m22.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m4.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "google-colab 1.0.0 requires pandas==1.5.3, but you have pandas 2.0.3 which is incompatible.\n",
            "pandas-gbq 0.17.9 requires pyarrow<10.0dev,>=3.0.0, but you have pyarrow 12.0.1 which is incompatible.\n",
            "tensorflow 2.12.0 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3, but you have protobuf 3.19.6 which is incompatible.\n",
            "tensorflow-datasets 4.9.2 requires protobuf>=3.20, but you have protobuf 3.19.6 which is incompatible.\n",
            "tensorflow-metadata 1.13.1 requires protobuf<5,>=3.20.3, but you have protobuf 3.19.6 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0m"
          ]
        }
      ],
      "source": [
        "!pip install -qU \\\n",
        "  pinecone-client==3.1.0 \\\n",
        "  pinecone-datasets==0.7.0 \\\n",
        "  pinecone-notebooks==0.1.1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Creating an Index\n",
        "\n",
        "We begin by initializing our connection to Pinecone. To do this we need a [free API key](https://app.pinecone.io)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "if not os.environ.get(\"PINECONE_API_KEY\"):\n",
        "    from pinecone_notebooks.colab import Authenticate\n",
        "    Authenticate()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from pinecone import Pinecone\n",
        "\n",
        "api_key = os.environ.get(\"PINECONE_API_KEY\")\n",
        "\n",
        "# configure client\n",
        "pc = Pinecone(api_key=api_key)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we setup our index specification, this allows us to define the cloud provider and region where we want to deploy our index. You can find a list of all [available providers and regions here](https://docs.pinecone.io/docs/projects)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from pinecone import ServerlessSpec\n",
        "\n",
        "cloud = os.environ.get('PINECONE_CLOUD') or 'aws'\n",
        "region = os.environ.get('PINECONE_REGION') or 'us-east-1'\n",
        "\n",
        "spec = ServerlessSpec(cloud=cloud, region=region)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "31jd7os7RAej",
        "outputId": "a5657975-ba80-43da-8c51-89812cb75d22"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['it-threats-fast']"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#List all present indexes associated with your key, should be empty on the first run\n",
        "pc.list_indexes().names()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "a7ysNAlrjD_k"
      },
      "source": [
        "## Installing other dependencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r3g-b61IywpQ",
        "outputId": "f973ecaa-132c-4115-f865-483739e5bccb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m578.0/578.0 MB\u001b[0m \u001b[31m1.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m10.8/10.8 MB\u001b[0m \u001b[31m41.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m50.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m42.6/42.6 kB\u001b[0m \u001b[31m3.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m5.9/5.9 MB\u001b[0m \u001b[31m64.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m438.7/438.7 kB\u001b[0m \u001b[31m23.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m4.9/4.9 MB\u001b[0m \u001b[31m42.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m781.3/781.3 kB\u001b[0m \u001b[31m26.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "pandas-gbq 0.17.9 requires pyarrow<10.0dev,>=3.0.0, but you have pyarrow 12.0.1 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0m"
          ]
        }
      ],
      "source": [
        "!pip install -qU \\\n",
        "  python-dateutil \\\n",
        "  tensorflow==2.10 \\\n",
        "  scikit-learn==1.3.0 \\\n",
        "  matplotlib==3.7.1"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "mc4ERmwniO1H"
      },
      "source": [
        "## Define a New Pinecone Index"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "9_TIgYxBywpV"
      },
      "outputs": [],
      "source": [
        "# Pick a name for the new service\n",
        "index_name = 'it-threats-fast'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "j7scTXFrywpV"
      },
      "outputs": [],
      "source": [
        "# Make sure index with the same name does not exist\n",
        "if index_name in pc.list_indexes().names():\n",
        "    pc.delete_index(index_name)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "wggPSrKtywpW"
      },
      "source": [
        "**Create an index**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "q5Q42v4mywpW"
      },
      "outputs": [],
      "source": [
        "pc.create_index(\n",
        "        index_name,\n",
        "        dimension=128,\n",
        "        metric='euclidean',\n",
        "        spec=spec\n",
        "    )"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "3eXIyr2-ywpW"
      },
      "source": [
        "**Connect to the index**\n",
        "\n",
        "We create an index object, a class instance of `pinecone.GRPCIndex`, which will be used to interact with the created index."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "tTlvU_jvywpW",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "index = pc.Index(index_name)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "IOP9jCo5ywpX"
      },
      "source": [
        "## Upload\n",
        "Here we load network events as precomputed vector embeddings from `pinecone-datasets` library, then upload them into Pinecone's vector index."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "PzkJJd8ZYTNM"
      },
      "source": [
        "### Download Data"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "vuOLNAehYTNN"
      },
      "source": [
        "**Download data for 22-02-2018 and 23-02-2018**\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "cGXf6PmhYTNR",
        "outputId": "8188949a-7f9b-4ca0-d151-deea2b45802e"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-0b9f6632-c877-40cf-9e2f-553dbf81fe43\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>values</th>\n",
              "      <th>sparse_values</th>\n",
              "      <th>metadata</th>\n",
              "      <th>blob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Ben_0</td>\n",
              "      <td>[0.0, 0.0, 0.0, 125628656.0, 0.0, 0.0, 5421442...</td>\n",
              "      <td>None</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Ben_1</td>\n",
              "      <td>[0.0, 0.0, 0.0, 356751744.0, 1190461440.0, 0.0...</td>\n",
              "      <td>None</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Ben_2</td>\n",
              "      <td>[0.0, 0.0, 0.0, 356751680.0, 1190461440.0, 0.0...</td>\n",
              "      <td>None</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Ben_3</td>\n",
              "      <td>[0.0, 0.0, 0.0, 125515856.0, 0.0, 0.0, 5432884...</td>\n",
              "      <td>None</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Ben_4</td>\n",
              "      <td>[0.0, 0.0, 0.0, 26214912.0, 698683840.0, 0.0, ...</td>\n",
              "      <td>None</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0b9f6632-c877-40cf-9e2f-553dbf81fe43')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-308e4096-91e5-46d2-8ce8-e56be58b8a08\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-308e4096-91e5-46d2-8ce8-e56be58b8a08')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-308e4096-91e5-46d2-8ce8-e56be58b8a08 button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-0b9f6632-c877-40cf-9e2f-553dbf81fe43 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-0b9f6632-c877-40cf-9e2f-553dbf81fe43');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "      id                                             values sparse_values  \\\n",
              "0  Ben_0  [0.0, 0.0, 0.0, 125628656.0, 0.0, 0.0, 5421442...          None   \n",
              "1  Ben_1  [0.0, 0.0, 0.0, 356751744.0, 1190461440.0, 0.0...          None   \n",
              "2  Ben_2  [0.0, 0.0, 0.0, 356751680.0, 1190461440.0, 0.0...          None   \n",
              "3  Ben_3  [0.0, 0.0, 0.0, 125515856.0, 0.0, 0.0, 5432884...          None   \n",
              "4  Ben_4  [0.0, 0.0, 0.0, 26214912.0, 698683840.0, 0.0, ...          None   \n",
              "\n",
              "              metadata  blob  \n",
              "0  {'label': 'Benign'}  None  \n",
              "1  {'label': 'Benign'}  None  \n",
              "2  {'label': 'Benign'}  None  \n",
              "3  {'label': 'Benign'}  None  \n",
              "4  {'label': 'Benign'}  None  "
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from pinecone_datasets import load_dataset\n",
        "\n",
        "dataset = load_dataset(\"it-threat-data-train\")\n",
        "dataset.head()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "sCkVwqm2KyM4"
      },
      "source": [
        "We don't need the `sparse_values` and `blob` columns so we drop:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "-psXytDLRAeo",
        "outputId": "4eddafd5-78c0-4b12-ff14-e2f012c5a3f1"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-9a89e026-f2e7-4e6b-8d2c-46bef4ae0df9\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>values</th>\n",
              "      <th>metadata</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Ben_0</td>\n",
              "      <td>[0.0, 0.0, 0.0, 125628656.0, 0.0, 0.0, 5421442...</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Ben_1</td>\n",
              "      <td>[0.0, 0.0, 0.0, 356751744.0, 1190461440.0, 0.0...</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Ben_2</td>\n",
              "      <td>[0.0, 0.0, 0.0, 356751680.0, 1190461440.0, 0.0...</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Ben_3</td>\n",
              "      <td>[0.0, 0.0, 0.0, 125515856.0, 0.0, 0.0, 5432884...</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Ben_4</td>\n",
              "      <td>[0.0, 0.0, 0.0, 26214912.0, 698683840.0, 0.0, ...</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9a89e026-f2e7-4e6b-8d2c-46bef4ae0df9')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-a76989fb-1070-4bfa-8a3d-2018da369c34\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-a76989fb-1070-4bfa-8a3d-2018da369c34')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-a76989fb-1070-4bfa-8a3d-2018da369c34 button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-9a89e026-f2e7-4e6b-8d2c-46bef4ae0df9 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-9a89e026-f2e7-4e6b-8d2c-46bef4ae0df9');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "      id                                             values  \\\n",
              "0  Ben_0  [0.0, 0.0, 0.0, 125628656.0, 0.0, 0.0, 5421442...   \n",
              "1  Ben_1  [0.0, 0.0, 0.0, 356751744.0, 1190461440.0, 0.0...   \n",
              "2  Ben_2  [0.0, 0.0, 0.0, 356751680.0, 1190461440.0, 0.0...   \n",
              "3  Ben_3  [0.0, 0.0, 0.0, 125515856.0, 0.0, 0.0, 5432884...   \n",
              "4  Ben_4  [0.0, 0.0, 0.0, 26214912.0, 698683840.0, 0.0, ...   \n",
              "\n",
              "              metadata  \n",
              "0  {'label': 'Benign'}  \n",
              "1  {'label': 'Benign'}  \n",
              "2  {'label': 'Benign'}  \n",
              "3  {'label': 'Benign'}  \n",
              "4  {'label': 'Benign'}  "
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "dataset.documents.drop(['sparse_values', 'blob'], axis=1, inplace=True)\n",
        "dataset.head()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "V3dx9XkPYTNV"
      },
      "source": [
        "### Upload Data"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "6C7er-8Gl2Rg"
      },
      "source": [
        "You can lower the `num_items` and, by doing so, limit the number of uploaded items."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "8iIbuW52zZVy",
        "outputId": "93d97fc3-b828-4809-a24e-f36992f5cd3d"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-3dbeb309-5fed-403b-a154-3e7ac85f1993\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>values</th>\n",
              "      <th>metadata</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Ben_0</td>\n",
              "      <td>[0.0, 0.0, 0.0, 125628656.0, 0.0, 0.0, 5421442...</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Ben_1</td>\n",
              "      <td>[0.0, 0.0, 0.0, 356751744.0, 1190461440.0, 0.0...</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Ben_2</td>\n",
              "      <td>[0.0, 0.0, 0.0, 356751680.0, 1190461440.0, 0.0...</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Ben_3</td>\n",
              "      <td>[0.0, 0.0, 0.0, 125515856.0, 0.0, 0.0, 5432884...</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Ben_4</td>\n",
              "      <td>[0.0, 0.0, 0.0, 26214912.0, 698683840.0, 0.0, ...</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3dbeb309-5fed-403b-a154-3e7ac85f1993')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-886db082-9a9f-451f-8330-4ff06e161b53\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-886db082-9a9f-451f-8330-4ff06e161b53')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-886db082-9a9f-451f-8330-4ff06e161b53 button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-3dbeb309-5fed-403b-a154-3e7ac85f1993 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-3dbeb309-5fed-403b-a154-3e7ac85f1993');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "      id                                             values  \\\n",
              "0  Ben_0  [0.0, 0.0, 0.0, 125628656.0, 0.0, 0.0, 5421442...   \n",
              "1  Ben_1  [0.0, 0.0, 0.0, 356751744.0, 1190461440.0, 0.0...   \n",
              "2  Ben_2  [0.0, 0.0, 0.0, 356751680.0, 1190461440.0, 0.0...   \n",
              "3  Ben_3  [0.0, 0.0, 0.0, 125515856.0, 0.0, 0.0, 5432884...   \n",
              "4  Ben_4  [0.0, 0.0, 0.0, 26214912.0, 698683840.0, 0.0, ...   \n",
              "\n",
              "              metadata  \n",
              "0  {'label': 'Benign'}  \n",
              "1  {'label': 'Benign'}  \n",
              "2  {'label': 'Benign'}  \n",
              "3  {'label': 'Benign'}  \n",
              "4  {'label': 'Benign'}  "
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "dataset.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 99,
          "referenced_widgets": [
            "d457fe5de4784553a9f47d7b4462ea4c",
            "cd5ed6888cab4656891522f1765eaf04",
            "ac390e5f1b9547318ab0478a86d0cf6b",
            "d0aeac530caa41a1b4c0ed56facfaefb",
            "da08a329e02147ca9f7029143682703f",
            "1c9e4ed606d1463e80f241e598b08fb8",
            "7cad2cfda84d4c1aa8898b8dff361a8b",
            "d3eb91a8850843b5833857901c1b76a3",
            "05b173e450494565b8d28c6585a527af",
            "f2d5170a93244298ba24b3f14b11f0c7",
            "da1de055c6834a82a69da37d5fa889e7",
            "80e09a9ff31d454fb6614f4923ff6433",
            "a6681a2392e04c91a77b8f8939c5bd7d",
            "e10a0235c9a949c8869b46b48c67b58c",
            "f52116b4976c4b1e8b408f09440ed52f",
            "0f90c5e13f034f85990c48026058ec32",
            "8e150825351a44dd94b792cefc96365c",
            "3ca5817c63cb4127b31f4973b7670388",
            "c53b3d1aaf524b62b293368c23ff358f",
            "9908e231310244d69be88a265e565be7",
            "a42a08cf00b140ebb4c883bde04febfb",
            "c0dec6e0d5ea4d09ae473aa9e498be1d"
          ]
        },
        "id": "_Ti9p0P-ywpX",
        "outputId": "728fc729-aa25-44ba-afd6-bb7e3a05b8aa",
        "scrolled": true
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d457fe5de4784553a9f47d7b4462ea4c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "sending upsert requests:   0%|          | 0/50000 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "80e09a9ff31d454fb6614f4923ff6433",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "collecting async responses:   0%|          | 0/100 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "upserted_count: 50000"
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "num_items = 50_000\n",
        "dataset = dataset.documents[:num_items]\n",
        "\n",
        "index.upsert_from_dataframe(dataset, batch_size=500)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "fglWJfAq_kw3"
      },
      "source": [
        "Let's verify all items were inserted."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xU172A4EywpY",
        "outputId": "14ef7372-44f8-4fcf-ea1e-7d41e5f181ad"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'dimension': 128,\n",
              " 'index_fullness': 0.5,\n",
              " 'namespaces': {'': {'vector_count': 50000}},\n",
              " 'total_vector_count': 50000}"
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "index.describe_index_stats()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "3CGzW3mVywpY"
      },
      "source": [
        "## Query"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Ywuld4BylAIu"
      },
      "source": [
        "First, we will randomly select a Benign/Attack event and query the vector index using the event embedding. Then, we will use data from different day, that contains same set of attacks to query on a bigger sample."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "tLs_wKDQRAeq"
      },
      "source": [
        "### Load the Model"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "NDLNS7zYRAeq"
      },
      "source": [
        "Here we load the pretrained model. The model is trained using the data from the same date."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "PzWle9Q9RAeq"
      },
      "source": [
        "We have modified [the original model](https://github.com/rambasnet/DeepLearning-IDS/blob/master/keras_tensorflow_models/02-23-2018.csv_adam_10_10_multiclass_baseline_model_1561316601.model) slightly and changed the number of classes from four (Benign, BruteForce-Web, BruteForce-XSS, SQL-Injection) to two (Benign and Attack). In the step below we will download and unzip our modified model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "jR6hucSjRAeq"
      },
      "outputs": [],
      "source": [
        "!wget -q -O it_threat_model.model.zip \"https://drive.google.com/uc?export=download&id=1VYMHOk_XMAc-QFJ_8CAPvWFfHnLpS2J_\"\n",
        "!unzip -q it_threat_model.model.zip"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "9zRyWjP8RAer"
      },
      "outputs": [],
      "source": [
        "from tensorflow import keras\n",
        "from tensorflow.keras.models import Model\n",
        "import tensorflow.keras.backend as K"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WPNez6x_RAer",
        "outputId": "28f36a01-d0ba-4abc-a88a-1c1268e2a10a"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named \"keras_metadata.pb\" in the SavedModel directory.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " dense (Dense)               (None, 128)               10240     \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 64)                8256      \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 1)                 65        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 18,561\n",
            "Trainable params: 18,561\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model = keras.models.load_model('it_threat_model.model')\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "qRouEveTRAer"
      },
      "outputs": [],
      "source": [
        "# Select the first layer\n",
        "layer_name = 'dense'\n",
        "intermediate_layer_model = Model(inputs=model.input,\n",
        "                                 outputs=model.get_layer(layer_name).output)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "m5H4rMyOYTNX"
      },
      "source": [
        "\n",
        "### Evaluate the Rare Event Classification Model"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "velMK_XlYTNX"
      },
      "source": [
        "We will use network intrusion precomputed dataset for 22-02-2018 for querying and testing the Pinecone."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "ajLLdSjRRAes",
        "outputId": "de277b3a-d1c9-450d-c61a-92bb0187794e"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-db92e0c8-69c7-4a37-aee2-51394b816bb5\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>values</th>\n",
              "      <th>sparse_values</th>\n",
              "      <th>metadata</th>\n",
              "      <th>blob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Ben_0</td>\n",
              "      <td>[0.0, 0.0, 0.0, 170976752.0, 187935920.0, 0.0,...</td>\n",
              "      <td>None</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Ben_1</td>\n",
              "      <td>[0.0, 0.0, 0.0, 211572592.0, 0.0, 0.0, 7663978...</td>\n",
              "      <td>None</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Ben_2</td>\n",
              "      <td>[0.0, 0.0, 0.0, 133747560.0, 770489344.0, 0.0,...</td>\n",
              "      <td>None</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Ben_3</td>\n",
              "      <td>[0.0, 0.0, 0.0, 133747288.0, 770489344.0, 0.0,...</td>\n",
              "      <td>None</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Ben_4</td>\n",
              "      <td>[0.0, 0.0, 0.0, 26222988.0, 698676736.0, 0.0, ...</td>\n",
              "      <td>None</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-db92e0c8-69c7-4a37-aee2-51394b816bb5')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-a2f459c9-b0e8-45c6-b2c2-89e38d4ae9ad\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-a2f459c9-b0e8-45c6-b2c2-89e38d4ae9ad')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-a2f459c9-b0e8-45c6-b2c2-89e38d4ae9ad button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-db92e0c8-69c7-4a37-aee2-51394b816bb5 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-db92e0c8-69c7-4a37-aee2-51394b816bb5');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "      id                                             values sparse_values  \\\n",
              "0  Ben_0  [0.0, 0.0, 0.0, 170976752.0, 187935920.0, 0.0,...          None   \n",
              "1  Ben_1  [0.0, 0.0, 0.0, 211572592.0, 0.0, 0.0, 7663978...          None   \n",
              "2  Ben_2  [0.0, 0.0, 0.0, 133747560.0, 770489344.0, 0.0,...          None   \n",
              "3  Ben_3  [0.0, 0.0, 0.0, 133747288.0, 770489344.0, 0.0,...          None   \n",
              "4  Ben_4  [0.0, 0.0, 0.0, 26222988.0, 698676736.0, 0.0, ...          None   \n",
              "\n",
              "              metadata  blob  \n",
              "0  {'label': 'Benign'}  None  \n",
              "1  {'label': 'Benign'}  None  \n",
              "2  {'label': 'Benign'}  None  \n",
              "3  {'label': 'Benign'}  None  \n",
              "4  {'label': 'Benign'}  None  "
            ]
          },
          "execution_count": 27,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "dataset_22 = load_dataset(\"it-threat-data-test\")\n",
        "dataset_22.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "in1nkunfRAes",
        "outputId": "33e76884-bd3f-4224-adf3-cd13a4dea35f"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-2504f67b-8ff1-4d51-b99f-3b718096ee5a\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>values</th>\n",
              "      <th>metadata</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Ben_0</td>\n",
              "      <td>[0.0, 0.0, 0.0, 170976752.0, 187935920.0, 0.0,...</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Ben_1</td>\n",
              "      <td>[0.0, 0.0, 0.0, 211572592.0, 0.0, 0.0, 7663978...</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Ben_2</td>\n",
              "      <td>[0.0, 0.0, 0.0, 133747560.0, 770489344.0, 0.0,...</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Ben_3</td>\n",
              "      <td>[0.0, 0.0, 0.0, 133747288.0, 770489344.0, 0.0,...</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Ben_4</td>\n",
              "      <td>[0.0, 0.0, 0.0, 26222988.0, 698676736.0, 0.0, ...</td>\n",
              "      <td>{'label': 'Benign'}</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2504f67b-8ff1-4d51-b99f-3b718096ee5a')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-54e873e9-0eea-4da0-b4a9-522912be0a49\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-54e873e9-0eea-4da0-b4a9-522912be0a49')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-54e873e9-0eea-4da0-b4a9-522912be0a49 button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-2504f67b-8ff1-4d51-b99f-3b718096ee5a button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-2504f67b-8ff1-4d51-b99f-3b718096ee5a');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "      id                                             values  \\\n",
              "0  Ben_0  [0.0, 0.0, 0.0, 170976752.0, 187935920.0, 0.0,...   \n",
              "1  Ben_1  [0.0, 0.0, 0.0, 211572592.0, 0.0, 0.0, 7663978...   \n",
              "2  Ben_2  [0.0, 0.0, 0.0, 133747560.0, 770489344.0, 0.0,...   \n",
              "3  Ben_3  [0.0, 0.0, 0.0, 133747288.0, 770489344.0, 0.0,...   \n",
              "4  Ben_4  [0.0, 0.0, 0.0, 26222988.0, 698676736.0, 0.0, ...   \n",
              "\n",
              "              metadata  \n",
              "0  {'label': 'Benign'}  \n",
              "1  {'label': 'Benign'}  \n",
              "2  {'label': 'Benign'}  \n",
              "3  {'label': 'Benign'}  \n",
              "4  {'label': 'Benign'}  "
            ]
          },
          "execution_count": 28,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "dataset_22.documents.drop(['sparse_values', 'blob'], axis=1, inplace=True)\n",
        "dataset_22.head()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "P6nJdtveYTNY"
      },
      "source": [
        "Let's define a sample that will include all different types of web attacks for this specific date."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N7vwZk6HYTNY",
        "outputId": "7ad7ec0d-f18b-4fba-c5b5-a2477a786a7a"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "metadata\n",
              "{'label': 'Benign'}              1638\n",
              "{'label': 'Brute Force -Web'}     249\n",
              "{'label': 'Brute Force -XSS'}      79\n",
              "{'label': 'SQL Injection'}         34\n",
              "Name: count, dtype: int64"
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "data_sample = dataset_22.documents[-2000:]\n",
        "data_sample.metadata.value_counts()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "neSxNwYckGMK"
      },
      "source": [
        "Now, we will query the test dataset and save predicted and expected results to create a confusion matrix."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "a654b98888a643f8b72d652c85a8fb6a",
            "62917e73c7eb48268506e5de72e6a233",
            "336b75c3eb5343dab90ac197784fe0a4",
            "5549d01ac4ac4041a0aa4ccaa7f3ee81",
            "1e482c1ab9ec47de9eb84f4abe544fcf",
            "59e9e2fd6b0b474d9d7f006a68b2e9ff",
            "cccfc5f59db740d689899d2fc783e4f4",
            "3a8bbd8bbc7044d2a54a4c2e877e2532",
            "c234a54fcaf14d0ba850eea647df6c56",
            "b3f2befb0abb42869f2e8066d2ae137c",
            "a95a29c73700484d9bb6dfcdc293490c"
          ]
        },
        "id": "8u6cg_1tYTNY",
        "outputId": "f8536aa0-e2e3-49ef-c8b7-09f3db977b6a",
        "scrolled": true
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a654b98888a643f8b72d652c85a8fb6a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "  0%|          | 0/20 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from tqdm.auto import tqdm\n",
        "from collections import Counter\n",
        "\n",
        "y_true = []\n",
        "y_pred = []\n",
        "\n",
        "BATCH_SIZE = 100\n",
        "\n",
        "for i in tqdm(range(0, len(data_sample), BATCH_SIZE)):\n",
        "    test_data = data_sample.iloc[i:i+BATCH_SIZE, :]\n",
        "    test_vector = test_data['values'].tolist()\n",
        "\n",
        "    # Query using the vector embedding\n",
        "    query_results = []\n",
        "\n",
        "    for xq in test_vector:\n",
        "        query_res = index.query(vector=xq.tolist(), top_k=50)\n",
        "        query_results.append(query_res)\n",
        "\n",
        "    ids = [res.id for result in query_results for res in result.matches]\n",
        "\n",
        "    for label, res in zip(test_data.metadata.values, query_results):\n",
        "        # Add to the true list\n",
        "        if label['label'] == 'Benign':\n",
        "            y_true.append(0)\n",
        "        else:\n",
        "            y_true.append(1)\n",
        "\n",
        "        counter = Counter(match.id.split('_')[0] for match in res.matches)\n",
        "\n",
        "        # Add to the predicted list\n",
        "        if counter['Bru'] or counter['SQL']:\n",
        "            y_pred.append(1)\n",
        "        else:\n",
        "            y_pred.append(0)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "id": "t5lG_ePFRAe9"
      },
      "outputs": [],
      "source": [
        "!pip install seaborn -qU"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "id": "dj2tAS6YRAe9"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import accuracy_score, precision_score, recall_score\n",
        "from sklearn.metrics import confusion_matrix\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 490
        },
        "id": "HV3-gkdWYTNZ",
        "outputId": "e7628098-81cc-4885-9d5c-4a74c15d360b"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[Text(0, 0.5, 'Benign'), Text(0, 1.5, 'Attack')]"
            ]
          },
          "execution_count": 33,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi8AAAHHCAYAAAB3K7g2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA23ElEQVR4nO3de3zP9f//8ft7mx3sYHMYU9owxhwjH6JyPpND5VjmXOoTFRIphz5RctanpBaSpNCSDsypIUo+DYk5nzJhzWEOb2yv3x9+3l/vRm3a9n4/1+16uexy6f18vV7P1+P1vrDunq/n8/WyWZZlCQAAwBAeri4AAAAgOwgvAADAKIQXAABgFMILAAAwCuEFAAAYhfACAACMQngBAABGIbwAAACjEF4AAIBRCC8A/tSePXvUrFkzFSpUSDabTXFxcTna/8GDB2Wz2TRnzpwc7ddkDRo0UIMGDVxdBuC2CC+AAfbt26fHH39cZcqUka+vr4KCglSvXj1NmzZNFy9ezNVzx8TEaPv27Xr11Vc1b9483XPPPbl6vrzUs2dP2Ww2BQUF3fR73LNnj2w2m2w2myZOnJjt/o8dO6bRo0crMTExB6oFcJ2XqwsA8Oe+/PJLPfLII/Lx8VGPHj1UuXJlXb58WevXr9fQoUO1Y8cOzZo1K1fOffHiRW3cuFEvvvii/v3vf+fKOcLDw3Xx4kUVKFAgV/r/K15eXrpw4YK++OILderUyWnb/Pnz5evrq0uXLt1W38eOHdOYMWMUERGh6tWrZ/m4FStW3Nb5gH8Kwgvgxg4cOKAuXbooPDxcq1evVlhYmGPbU089pb179+rLL7/MtfOfPHlSkhQcHJxr57DZbPL19c21/v+Kj4+P6tWrpwULFmQKLx999JFat26txYsX50ktFy5cUMGCBeXt7Z0n5wNMxW0jwI1NmDBBaWlpio2NdQou10VGRmrQoEGOz1evXtUrr7yismXLysfHRxERERoxYoTsdrvTcREREWrTpo3Wr1+vf/3rX/L19VWZMmX0wQcfOPYZPXq0wsPDJUlDhw6VzWZTRESEpGu3W67/941Gjx4tm83m1BYfH6/77rtPwcHBCggIUFRUlEaMGOHYfqs5L6tXr9b9998vf39/BQcHq127dtq5c+dNz7d371717NlTwcHBKlSokHr16qULFy7c+ov9g27duunrr7/W6dOnHW2bN2/Wnj171K1bt0z7//777xoyZIiqVKmigIAABQUFqWXLltq6datjn7Vr16pWrVqSpF69ejluP12/zgYNGqhy5crasmWLHnjgARUsWNDxvfxxzktMTIx8fX0zXX/z5s0VEhKiY8eOZflagfyA8AK4sS+++EJlypRR3bp1s7R/37599fLLL6tGjRqaMmWK6tevr/Hjx6tLly6Z9t27d68efvhhNW3aVJMmTVJISIh69uypHTt2SJI6duyoKVOmSJK6du2qefPmaerUqdmqf8eOHWrTpo3sdrvGjh2rSZMm6cEHH9SGDRv+9LiVK1eqefPmOnHihEaPHq3nnntO3333nerVq6eDBw9m2r9Tp046d+6cxo8fr06dOmnOnDkaM2ZMluvs2LGjbDablixZ4mj76KOPVKFCBdWoUSPT/vv371dcXJzatGmjyZMna+jQodq+fbvq16/vCBIVK1bU2LFjJUn9+/fXvHnzNG/ePD3wwAOOflJSUtSyZUtVr15dU6dOVcOGDW9a37Rp01SsWDHFxMQoPT1dkvTOO+9oxYoVmjFjhkqWLJnlawXyBQuAWzpz5owlyWrXrl2W9k9MTLQkWX379nVqHzJkiCXJWr16taMtPDzckmQlJCQ42k6cOGH5+PhYgwcPdrQdOHDAkmS98cYbTn3GxMRY4eHhmWoYNWqUdeOvlSlTpliSrJMnT96y7uvnmD17tqOtevXqVmhoqJWSkuJo27p1q+Xh4WH16NEj0/l69+7t1GeHDh2sIkWK3PKcN16Hv7+/ZVmW9fDDD1uNGze2LMuy0tPTrRIlSlhjxoy56Xdw6dIlKz09PdN1+Pj4WGPHjnW0bd68OdO1XVe/fn1LkjVz5sybbqtfv75T2/Llyy1J1n/+8x9r//79VkBAgNW+ffu/vEYgP2LkBXBTZ8+elSQFBgZmaf+vvvpKkvTcc885tQ8ePFiSMs2NiY6O1v333+/4XKxYMUVFRWn//v23XfMfXZ8r8/nnnysjIyNLxyQnJysxMVE9e/ZU4cKFHe1Vq1ZV06ZNHdd5oyeeeMLp8/3336+UlBTHd5gV3bp109q1a3X8+HGtXr1ax48fv+ktI+naPBkPj2u/PtPT05WSkuK4Jfa///0vy+f08fFRr169srRvs2bN9Pjjj2vs2LHq2LGjfH199c4772T5XEB+QngB3FRQUJAk6dy5c1na/9ChQ/Lw8FBkZKRTe4kSJRQcHKxDhw45td91112Z+ggJCVFqauptVpxZ586dVa9ePfXt21fFixdXly5d9Mknn/xpkLleZ1RUVKZtFStW1KlTp3T+/Hmn9j9eS0hIiCRl61patWqlwMBALVy4UPPnz1etWrUyfZfXZWRkaMqUKSpXrpx8fHxUtGhRFStWTNu2bdOZM2eyfM477rgjW5NzJ06cqMKFCysxMVHTp09XaGholo8F8hPCC+CmgoKCVLJkSf3888/ZOu6PE2ZvxdPT86btlmXd9jmuz8e4zs/PTwkJCVq5cqUee+wxbdu2TZ07d1bTpk0z7ft3/J1ruc7Hx0cdO3bU3Llz9dlnn91y1EWSxo0bp+eee04PPPCAPvzwQy1fvlzx8fGqVKlSlkeYpGvfT3b89NNPOnHihCRp+/bt2ToWyE8IL4Aba9Omjfbt26eNGzf+5b7h4eHKyMjQnj17nNp/++03nT592rFyKCeEhIQ4rcy57o+jO5Lk4eGhxo0ba/Lkyfrll1/06quvavXq1VqzZs1N+75eZ1JSUqZtu3btUtGiReXv7//3LuAWunXrpp9++knnzp276STn6xYtWqSGDRsqNjZWXbp0UbNmzdSkSZNM30lWg2RWnD9/Xr169VJ0dLT69++vCRMmaPPmzTnWP2ASwgvgxp5//nn5+/urb9+++u233zJt37dvn6ZNmybp2m0PSZlWBE2ePFmS1Lp16xyrq2zZsjpz5oy2bdvmaEtOTtZnn33mtN/vv/+e6djrD2v74/Lt68LCwlS9enXNnTvXKQz8/PPPWrFiheM6c0PDhg31yiuv6M0331SJEiVuuZ+np2emUZ1PP/1Uv/76q1Pb9ZB1s6CXXcOGDdPhw4c1d+5cTZ48WREREYqJibnl9wjkZzykDnBjZcuW1UcffaTOnTurYsWKTk/Y/e677/Tpp5+qZ8+ekqRq1aopJiZGs2bN0unTp1W/fn398MMPmjt3rtq3b3/LZbi3o0uXLho2bJg6dOiggQMH6sKFC3r77bdVvnx5pwmrY8eOVUJCglq3bq3w8HCdOHFCb731lu68807dd999t+z/jTfeUMuWLXXvvfeqT58+unjxombMmKFChQpp9OjROXYdf+Th4aGRI0f+5X5t2rTR2LFj1atXL9WtW1fbt2/X/PnzVaZMGaf9ypYtq+DgYM2cOVOBgYHy9/dX7dq1Vbp06WzVtXr1ar311lsaNWqUY+n27Nmz1aBBA7300kuaMGFCtvoDjOfi1U4AsmD37t1Wv379rIiICMvb29sKDAy06tWrZ82YMcO6dOmSY78rV65YY8aMsUqXLm0VKFDAKlWqlDV8+HCnfSzr2lLp1q1bZzrPH5fo3mqptGVZ1ooVK6zKlStb3t7eVlRUlPXhhx9mWiq9atUqq127dlbJkiUtb29vq2TJklbXrl2t3bt3ZzrHH5cTr1y50qpXr57l5+dnBQUFWW3btrV++eUXp32un++PS7Fnz55tSbIOHDhwy+/UspyXSt/KrZZKDx482AoLC7P8/PysevXqWRs3brzpEufPP//cio6Otry8vJyus379+lalSpVues4b+zl79qwVHh5u1ahRw7py5YrTfs8++6zl4eFhbdy48U+vAchvbJaVjRltAAAALsacFwAAYBTCCwAAMArhBQAAGIXwAgAAjEJ4AQAARiG8AAAAoxBeAACAUfLlE3b97v63q0sAkEtSN7/p6hIA5BLfLKYSRl4AAIBRCC8AAMAohBcAAGAUwgsAADAK4QUAABiF8AIAAIxCeAEAAEYhvAAAAKMQXgAAgFEILwAAwCiEFwAAYBTCCwAAMArhBQAAGIXwAgAAjEJ4AQAARiG8AAAAoxBeAACAUQgvAADAKIQXAABgFMILAAAwCuEFAAAYhfACAACMQngBAABGIbwAAACjEF4AAIBRCC8AAMAohBcAAGAUwgsAADAK4QUAABiF8AIAAIxCeAEAAEYhvAAAAKMQXgAAgFEILwAAwCiEFwAAYBTCCwAAMArhBQAAGIXwAgAAjEJ4AQAARiG8AAAAoxBeAACAUQgvAADAKIQXAABgFMILAAAwCuEFAAAYhfACAACMQngBAABGIbwAAACjEF4AAIBRCC8AAMAohBcAAGAUwgsAADAK4QUAABiF8AIAAIxCeAEAAEYhvAAAAKMQXgAAgFEILwAAwCiEFwAAYBTCCwAAMArhBQAAGIXwAgAAjEJ4AQAARiG8AAAAoxBeAACAUQgvAADAKIQXAABgFMILAAAwCuEFAAAYhfACAACMQngBAABG8XJ1AddlZGRo7969OnHihDIyMpy2PfDAAy6qCgAAuBu3CC+bNm1St27ddOjQIVmW5bTNZrMpPT3dRZUBAAB34xbh5YknntA999yjL7/8UmFhYbLZbK4uCQAAuCm3CC979uzRokWLFBkZ6epSAACAm3OLCbu1a9fW3r17XV0GAAAwgFuMvDz99NMaPHiwjh8/ripVqqhAgQJO26tWreqiygAAgLuxWX+cIesCHh6ZB4BsNpssy7qtCbt+d/87p0oD4GZSN7/p6hIA5BLfLA6puMXIy4EDB1xdAgAAMIRbhJfw8HBXlwAAAAzhFuFl6dKlN2232Wzy9fVVZGSkSpcuncdVAQAAd+QW4aV9+/aOOS43unHey3333ae4uDiFhIS4qEoAAOAO3GKpdHx8vGrVqqX4+HidOXNGZ86cUXx8vGrXrq1ly5YpISFBKSkpGjJkiKtLBQAALuYWIy+DBg3SrFmzVLduXUdb48aN5evrq/79+2vHjh2aOnWqevfu7cIqAQCAO3CLkZd9+/YpKCgoU3tQUJD2798vSSpXrpxOnTqV16UBAAA34xbhpWbNmho6dKhOnjzpaDt58qSef/551apVS9K1VwiUKlXKVSUCAAA34Ra3jWJjY9WuXTvdeeedjoBy5MgRlSlTRp9//rkkKS0tTSNHjnRlmcgl9WqU1bM9mqhG9F0KK1ZInZ6dpS/WbnPaJ6p0cf1nUHvdXyNSXl4e2rX/uLoOeU9HjqdKkma82EWNakcprFghpV20a9PWAxo57XPtPvibo4+a0XfplYHtdHd0KVmW9OPPh/TitDht3/1rnl4vgKz5+KP5mjs7VqdOnVT5qAp6YcRLqsIT1yE3ecKuJGVkZGjFihXavXu3JCkqKkpNmza96dN3/wpP2DVLs3rRurdaGf2087AWTu6fKbyUvrOo1s0bqrlx3+mTb7bo7PlLii4bph+2HdDJ1DRJUu+O9ZR08LiOJKeqcKGCevGJ1qpW/g5VaDNKGRmW/P28lfTVK/ry2+2aOHuFvDw99NKA1rq3elmVazlSV69muOrykU08Yfef4Zuvv9LI4c9r5KgxqlKlmubPm6sVK77R58u+UZEiRVxdHnJJVp+w6zbhJScRXsx18ac3M4WXD17rpStX0tXnpQ+y3E/lciW1+ZMRim47WgeOnlKN6Lu0Yf7zKtdipI7+dlqSVCmypH78dIQqPTha+48wn8oUhJd/hu5dHlGlylU0YuTLkq79A7dZ4/rq2u0x9enX38XVIbe4/esBpk+frv79+8vX11fTp0//030HDhyYR1XB3dhsNrW4r5Imz12ppf99StUq3KlDv6bojfdXZLq1dF1BX2/1eLCODhw9paP//7bS7oO/6VRqmmLa19WE2OXy9PRQz/b3auf+ZB069nteXhKAv3Dl8mXt/GWH+vR73NHm4eGhOnXqatvWn1xYGdyFy8LLlClT1L17d/n6+mrKlCm33M9msxFe/sFCCwco0N9XQ3o11Zj/LtPIaXFqVi9aH0/qq+b9p2v9lr2Offs/cr9efaa9Agr6KOnAcbUe8KauXL32Us+0C3Y17zdNn0zur+H9WkiS9h4+oQef+q/S07llBLiT1NOpSk9Pz3R7qEiRIjpwYL+LqoI7cVl4ufFljH/nxYx2u112u92pzcpIl83D87b7hPu4Pudp2drtmjF/jSRp2+5fVbtaGfV7+D6n8PLx15u16vtdKlE0SM/0aKIPX++tRr0my375qnx9CmjmqO7auHW/YobPlqenh57p0VhLpg/QfY++oUv2Ky65PgBA9rnFUum/Y/z48SpUqJDTz9Xftri6LOSQU6lpunIlXTv3Jzu1J+0/rlIlnF8VcTbtkvYdPqkN/9unbkPeU1Tp4mrXqJokqXPLe3RXycLqP+pDbfnlsH7YflAxw+co4o4iatuA1QuAOwkJDpGnp6dSUlKc2lNSUlS0aFEXVQV34hZLpdPT0zVnzhytWrVKJ06cUEaG8zD+6tWrb3ns8OHD9dxzzzm1hd4/LFfqRN67cjVdW345pPLhxZ3ay4WH6nBy6i2Ps9lssskm7wLX/ogX9PVWRobl9P6sDMuSZUkeNlvuFA/gthTw9lbF6Er6ftNGNWrcRNK1Cbvff79RXbo+6uLq4A7cIrwMGjRIc+bMUevWrVW5cmXZsvE/Ex8fH/n4+Di1ccvILP5+3ipbqpjjc8QdRVS1/B1KPXtBR46nasrclZr3em+t/99effvjbjWrG61WD1RW837THPs/3LymVm3cqVOpabqjeLAG92qmi/YrWr5+hyRp1aZdGvdMe00d3klvf/ytPGw2DenVTFfT0/Xtj7tdct0Abu2xmF56acQwVapUWZWrVNWH8+bq4sWLat+ho6tLgxtwi6XSRYsW1QcffKBWrVrlSH8slTbL/TXLacV7gzK1z1u6Sf1HfShJ6tGujob2bqY7QoO1+9AJ/Wfml1q2drskKaxYIb31cjfdXbGUQoIK6kTKOa3/316Nm/W19hw64eivUe0KevHxloqODFNGhqWtu45q9H+/0A/bD+bJdSJnsFT6n2PB/A8dD6mLqlBRw0aMVNWq1VxdFnKRUc95KVmypNauXavy5cvnSH+EFyD/IrwA+VdWw4tbTNgdPHiwpk2bJjfIUQAAwM25xZyX9evXa82aNfr6669VqVIlFShQwGn7kiVLXFQZAABwN24RXoKDg9WhQwdXlwEAAAzgFuFl9uzZri4BAAAYwi3mvEjS1atXtXLlSr3zzjs6d+6cJOnYsWNKS0tzcWUAAMCduMXIy6FDh9SiRQsdPnxYdrtdTZs2VWBgoF5//XXZ7XbNnDnT1SUCAAA34RYjL4MGDdI999yj1NRU+fn5Odo7dOigVatWubAyAADgbtxi5GXdunX67rvv5O3t7dQeERGhX3/91UVVAQAAd+QWIy8ZGRlKT0/P1H706FEFBga6oCIAAOCu3CK8NGvWTFOnTnV8ttlsSktL06hRo3LslQEAACB/cIvXAxw9elTNmzeXZVnas2eP7rnnHu3Zs0dFihTRunXrFBoamq3+eD0AkH/xegAg/zLq3UbStaXSH3/8sbZt26a0tDTVqFFD3bt3d5rAm1WEFyD/IrwA+ZdR7zZKSUmRl5eXHn30UT399NMqWrSokpKS9OOPP7q6NAAA4GZcGl62b9+uiIgIhYaGqkKFCkpMTFStWrU0ZcoUzZo1Sw0bNlRcXJwrSwQAAG7GpeHl+eefV5UqVZSQkKAGDRqoTZs2at26tc6cOaPU1FQ9/vjjeu2111xZIgAAcDMunfNStGhRrV69WlWrVlVaWpqCgoK0efNm1axZU5K0a9cu1alTR6dPn85Wv8x5AfIv5rwA+ZcRc15+//13lShRQpIUEBAgf39/hYSEOLaHhIQ43nMEAAAgucGEXZvN9qefAQAAbuTy1wP07NlTPj4+kqRLly7piSeekL+/vyTJbre7sjQAAOCGXBpeYmJinD4/+uijmfbp0aNHXpUDAAAM4NLwMnv2bFeeHgAAGMjlc14AAACyg/ACAACMQngBAABGIbwAAACjEF4AAIBRCC8AAMAohBcAAGAUwgsAADAK4QUAABiF8AIAAIxCeAEAAEYhvAAAAKMQXgAAgFEILwAAwCiEFwAAYBTCCwAAMArhBQAAGIXwAgAAjEJ4AQAARiG8AAAAoxBeAACAUQgvAADAKIQXAABgFMILAAAwCuEFAAAYhfACAACMQngBAABGIbwAAACjEF4AAIBRCC8AAMAohBcAAGAUwgsAADAK4QUAABiF8AIAAIxCeAEAAEYhvAAAAKMQXgAAgFEILwAAwCiEFwAAYBTCCwAAMArhBQAAGIXwAgAAjEJ4AQAARiG8AAAAoxBeAACAUQgvAADAKIQXAABgFK+s7LR06dIsd/jggw/edjEAAAB/JUvhpX379lnqzGazKT09/e/UAwAA8KeyFF4yMjJyuw4AAIAsYc4LAAAwSpZGXv7o/Pnz+vbbb3X48GFdvnzZadvAgQNzpDAAAICbyXZ4+emnn9SqVStduHBB58+fV+HChXXq1CkVLFhQoaGhhBcAAJCrsn3b6Nlnn1Xbtm2VmpoqPz8/bdq0SYcOHVLNmjU1ceLE3KgRAADAIdvhJTExUYMHD5aHh4c8PT1lt9tVqlQpTZgwQSNGjMiNGgEAAByyHV4KFCggD49rh4WGhurw4cOSpEKFCunIkSM5Wx0AAMAfZHvOy913363NmzerXLlyql+/vl5++WWdOnVK8+bNU+XKlXOjRgAAAIdsj7yMGzdOYWFhkqRXX31VISEhGjBggE6ePKlZs2bleIEAAAA3slmWZbm6iJzmd/e/XV0CgFySuvlNV5cAIJf4ZvF+EA+pAwAARsn2nJfSpUvLZrPdcvv+/fv/VkEAAAB/Jtvh5ZlnnnH6fOXKFf3000/65ptvNHTo0JyqCwAA4KayHV4GDRp00/b//ve/+vHHH/92QQAAAH8mx+a8tGzZUosXL86p7gAAAG4qx8LLokWLVLhw4ZzqDgAA4KZu6yF1N07YtSxLx48f18mTJ/XWW2/laHEAAAB/lO3w0q5dO6fw4uHhoWLFiqlBgwaqUKFCjhZ3u9Z/Ns7VJQAAgFySLx9St+XgWVeXACCXVLozyNUlAMglufaQOk9PT504cSJTe0pKijw9PbPbHQAAQLZkO7zcaqDGbrfL29v7bxcEAADwZ7I852X69OmSJJvNpvfee08BAQGObenp6UpISHCbOS8AACD/ynJ4mTJliqRrIy8zZ850ukXk7e2tiIgIzZw5M+crBAAAuEGWw8uBAwckSQ0bNtSSJUsUEhKSa0UBAADcSraXSq9ZsyY36gAAAMiSbE/Yfeihh/T6669nap8wYYIeeeSRHCkKAADgVrIdXhISEtSqVatM7S1btlRCQkKOFAUAAHAr2Q4vaWlpN10SXaBAAZ09y8PhAABA7sp2eKlSpYoWLlyYqf3jjz9WdHR0jhQFAABwK9mesPvSSy+pY8eO2rdvnxo1aiRJWrVqlT766CMtWrQoxwsEAAC4UbbDS9u2bRUXF6dx48Zp0aJF8vPzU7Vq1bR69WoVLlw4N2oEAABw+NsvZjx79qwWLFig2NhYbdmyRenp6TlV223jxYxA/sWLGYH8K9dezHhdQkKCYmJiVLJkSU2aNEmNGjXSpk2bbrc7AACALMnWbaPjx49rzpw5io2N1dmzZ9WpUyfZ7XbFxcUxWRcAAOSJLI+8tG3bVlFRUdq2bZumTp2qY8eOacaMGblZGwAAQCZZHnn5+uuvNXDgQA0YMEDlypXLzZoAAABuKcsjL+vXr9e5c+dUs2ZN1a5dW2+++aZOnTqVm7UBAABkkuXwUqdOHb377rtKTk7W448/ro8//lglS5ZURkaG4uPjde7cudysEwAAQNLfXCqdlJSk2NhYzZs3T6dPn1bTpk21dOnSnKzvtrBUGsi/WCoN5F+5vlRakqKiojRhwgQdPXpUCxYs+DtdAQAAZMnffkidO2LkBci/GHkB8q88GXkBAADIa4QXAABgFMILAAAwCuEFAAAYhfACAACMQngBAABGIbwAAACjEF4AAIBRCC8AAMAohBcAAGAUwgsAADAK4QUAABiF8AIAAIxCeAEAAEYhvAAAAKMQXgAAgFEILwAAwCiEFwAAYBTCCwAAMArhBQAAGIXwAgAAjEJ4AQAARiG8AAAAoxBeAACAUQgvAADAKIQXAABgFMILAAAwCuEFAAAYhfACAACMQngBAABGIbwAAACjEF4AAIBRCC8AAMAohBcAAGAUwgsAADAK4QUAABiF8AIAAIxCeAEAAEYhvAAAAKMQXgAAgFEILwAAwCiEFwAAYBTCCwAAMArhBQAAGIXwAgAAjEJ4AQAARiG8AAAAoxBeAACAUVweXs6ePXvLbXv37s3DSgAAgAlcHl5at24tu92eqT0pKUkNGjTI+4IAAIBbc3l4CQgIUIcOHXT16lVH286dO9WgQQM99NBDLqwMAAC4I5eHlyVLlujMmTPq3r27LMvSzz//rAYNGqhr166aNm2aq8sDAABuxmZZluXqIk6fPq0GDRqoXLlySkhIUI8ePfTGG2/cdn9bDt56Hg0As1W6M8jVJQDIJb5eWdvPJeHlZpN0k5OT1bRpU7Vp00avvfaaoz0oKPu/qAgvQP5FeAHyL7cOLx4eHrLZbJnar5dis9lkWZZsNpvS09Oz3T/hBci/CC9A/pXV8JLF3XLWmjVrXHFaAACQD7gkvNSvX98VpwUAAPmAy1cbzZ49W59++mmm9k8//VRz5851QUUAAMCduTy8jB8/XkWLFs3UHhoaqnHjxrmgIgAA4M5cHl4OHz6s0qVLZ2oPDw/X4cOHXVARAABwZy4PL6Ghodq2bVum9q1bt6pIkSIuqAgAALgzl4eXrl27auDAgVqzZo3S09OVnp6u1atXa9CgQerSpYurywMAAG7GJauNbvTKK6/o4MGDaty4sby8rpWTkZGhHj16MOcFAABk4havB5Ck3bt3a+vWrfLz81OVKlUUHh5+233xkDqzxX+xSCu/XKxTvyVLku4IL6OO3fuoeq16jn12/7JNn8x5W/t2/SwPT0+FlymvF8ZNl7ePr04eP6bPPorVjsQfdTo1RSFFiuq+Ri3VvmtveRUo4KrLQg7hIXX/HB9/NF9zZ8fq1KmTKh9VQS+MeElVqlZ1dVnIRW79hN3cRngx25ZNCfLw8FSJO0pJlqWE+C+1bNE8jf/vh7ozoqx2/7JNr784UO269FSN2vfLw9NTh/fvUc1766uAt7e2bv5OG7+NV92GzVW85J06enCf3p06Tvc3bqnu/Z9x9eXhbyK8/DN88/VXGjn8eY0cNUZVqlTT/HlztWLFN/p82TfMh8zHjAovR48e1dKlS3X48GFdvnzZadvkyZOz3R/hJf/p91Bjdes3UA1btNPLg3qpco1/qVPMgCwf/8Wn87Ry2SJNm/t5LlaJvEB4+Wfo3uURVapcRSNGvizp2nSCZo3rq2u3x9SnX38XV4fc4tavB7jRqlWr9OCDD6pMmTLatWuXKleurIMHD8qyLNWoUcPV5cHFMtLTtWndKtntF1WuYhWdOf279u76WfUatdCoZ3rrt+RfVbJUuDr1fFIVKle/ZT8Xz6cpILBQ3hUO4LZduXxZO3/ZoT79Hne0eXh4qE6dutq29ScXVgZ34fLVRsOHD9eQIUO0fft2+fr6avHixTpy5Ijq16+vRx555C+Pt9vtOnv2rNPPZbs9DypHbjp8YK96tXtAPdrU0/vTx+vZl9/QneFldCL5V0nS4nnvqmHL9nrh1ekqHVlB4154Usm/3vy5QMd/PaLlny9U41Yd8vISANym1NOpSk9Pz3R7qEiRIjp16pSLqoI7cXl42blzp3r06CFJ8vLy0sWLFxUQEKCxY8fq9ddf/8vjx48fr0KFCjn9zH47+7ea4F5K3hmu8W/N19jps9WkzUOaOXG0jh7aLysjQ5LUqFUHNWj+oCIio/TYE88p7M5wfbt8aaZ+fj91Qq+/OFC1H2iiRoQXAMgXXH7byN/f3zHPJSwsTPv27VOlSpUkKUsJe/jw4Xruueec2nYkM/JiOq8CBa5N2JVUplxF7Uv6Rd/EfawHO8dIku4Md34q8x2lInTqxHGnttSUk/rP8wNULrqq+g4akTeFA/jbQoJD5OnpqZSUFKf2lJSUm75OBv88Lh95qVOnjtavXy9JatWqlQYPHqxXX31VvXv3Vp06df7yeB8fHwUFBTn9ePv45HbZyGOWZenqlcsqVrykQooU07Gjh5y2J/96WEVDwxyffz91Qq8MfUKly1XQE4NfloeHy/+oA8iiAt7eqhhdSd9v2uhoy8jI0Pffb1TVane7sDK4C5ePvEyePFlpaWmSpDFjxigtLU0LFy5UuXLlbmulEcz38ftvqlqtuiparIQuXryg79Z8o53btuiFV2fIZrOpzcOPatG8WQovU17hZcorYeUyHTtySM+MvHab8XpwKRpaQt37DdLZM6mOvoML8682wASPxfTSSyOGqVKlyqpcpao+nDdXFy9eVPsOHV1dGtyAWyyVzmkslTbbrMmv6OfEzTr9+ykVLBigUqUj9WCnGFWpWduxz9KFc7Ri6ac6f+6s7ipTTl37DnSsNvp2xRd6Z9LYm/b90fLNeXEJyEUslf7nWDD/Q8dD6qIqVNSwESNVtWo1V5eFXGTMc17KlCmjzZs3Z5pVfvr0adWoUUP79+/Pdp+EFyD/IrwA+VdWw4vLJwIcPHhQ6enpmdrtdrt+/fVXF1QEAADcmcvmvCxd+n/LWpcvX65Chf7vAWLp6elatWqVIiIiXFAZAABwZy67bfRnqz8KFCigiIgITZo0SW3atMl239w2AvIvbhsB+Zfbvx4g4/8/bKx06dLavHkza/cBAECWuHzOy5gxYxQYGJip/fLly/rggw9cUBEAAHBnLl9t5OnpqeTkZIWGhjq1p6SkKDQ09KaTef8Kt42A/IvbRkD+ZcxqI8uyZLPZMrUfPXrUaRIvAACA5MI5L3fffbdsNptsNpsaN24sL6//KyU9PV0HDhxQixYtXFUeAABwUy4LL+3bt5ckJSYmqnnz5goICHBs8/b2VkREhCpXruyi6gAAgLty+ZyXuXPnqnPnzvL19ZUknTt3TgsWLNB7772nLVu2MOcFgBPmvAD5lzFzXmJiYuTr66uEhATFxMQoLCxMEydOVKNGjbRp0yZXlwcAANyMS98qffz4cc2ZM0exsbE6e/asOnXqJLvdrri4OEVHR7uyNAAA4KZcNvLStm1bRUVFadu2bZo6daqOHTumGTNmuKocAABgCJeNvHz99dcaOHCgBgwYoHLlyrmqDAAAYBiXjbysX79e586dU82aNVW7dm29+eabOnXqlKvKAQAAhnBZeKlTp47effddJScn6/HHH9fHH3+skiVLKiMjQ/Hx8Tp37pyrSgMAAG7M5Uulb5SUlKTY2FjNmzdPp0+fVtOmTbV06dJs98NSaSD/Yqk0kH8Zs1T6RlFRUZowYYKOHj2qBQsWuLocAADghtxq5CWnMPIC5F+MvAD5l5EjLwAAAH+F8AIAAIxCeAEAAEYhvAAAAKMQXgAAgFEILwAAwCiEFwAAYBTCCwAAMArhBQAAGIXwAgAAjEJ4AQAARiG8AAAAoxBeAACAUQgvAADAKIQXAABgFMILAAAwCuEFAAAYhfACAACMQngBAABGIbwAAACjEF4AAIBRCC8AAMAohBcAAGAUwgsAADAK4QUAABiF8AIAAIxCeAEAAEYhvAAAAKMQXgAAgFEILwAAwCiEFwAAYBTCCwAAMArhBQAAGIXwAgAAjEJ4AQAARiG8AAAAoxBeAACAUQgvAADAKIQXAABgFMILAAAwCuEFAAAYhfACAACMQngBAABGIbwAAACjEF4AAIBRCC8AAMAohBcAAGAUwgsAADAK4QUAABiF8AIAAIxCeAEAAEYhvAAAAKMQXgAAgFEILwAAwCiEFwAAYBTCCwAAMArhBQAAGIXwAgAAjEJ4AQAARiG8AAAAoxBeAACAUQgvAADAKIQXAABgFMILAAAwCuEFAAAYhfACAACMQngBAABGIbwAAACj2CzLslxdBHC77Ha7xo8fr+HDh8vHx8fV5QDIQfz9xq0QXmC0s2fPqlChQjpz5oyCgoJcXQ6AHMTfb9wKt40AAIBRCC8AAMAohBcAAGAUwguM5uPjo1GjRjGZD8iH+PuNW2HCLgAAMAojLwAAwCiEFwAAYBTCCwAAMArhBflKRESEpk6d6uoyALjY2rVrZbPZdPr0aVeXglxAeEGe6Nmzp2w2m+OnSJEiatGihbZt25aj59m8ebP69++fo30CyJ6NGzfK09NTrVu3dmofPXq0qlevnml/m82muLi4vCkO+QLhBXmmRYsWSk5OVnJyslatWiUvLy+1adMmR89RrFgxFSxYMEf7BJA9sbGxevrpp5WQkKBjx465uhzkQ4QX5BkfHx+VKFFCJUqUUPXq1fXCCy/oyJEjOnnypCTpyJEj6tSpk4KDg1W4cGG1a9dOBw8edBzfs2dPtW/fXhMnTlRYWJiKFCmip556SleuXHHs88fbRrt27dJ9990nX19fRUdHa+XKlU7/yjt48KBsNpuWLFmihg0bqmDBgqpWrZo2btyYF18JkO+kpaVp4cKFGjBggFq3bq05c+ZIkubMmaMxY8Zo69atjhHYOXPmKCIiQpLUoUMH2Ww2x+d9+/apXbt2Kl68uAICAlSrVi2tXLnS6Vx2u13Dhg1TqVKl5OPjo8jISMXGxt60rgsXLqhly5aqV68et5LyAcILXCItLU0ffvihIiMjVaRIEV25ckXNmzdXYGCg1q1bpw0bNiggIEAtWrTQ5cuXHcetWbNG+/bt05o1azR37lzNmTPH8cvxj9LT09W+fXsVLFhQ33//vWbNmqUXX3zxpvu++OKLGjJkiBITE1W+fHl17dpVV69ezY1LB/K1Tz75RBUqVFBUVJQeffRRvf/++7IsS507d9bgwYNVqVIlxwhs586dtXnzZknS7NmzlZyc7PiclpamVq1aadWqVfrpp5/UokULtW3bVocPH3acq0ePHlqwYIGmT5+unTt36p133lFAQECmmk6fPq2mTZsqIyND8fHxCg4OzpPvArnIAvJATEyM5enpafn7+1v+/v6WJCssLMzasmWLZVmWNW/ePCsqKsrKyMhwHGO32y0/Pz9r+fLljj7Cw8Otq1evOvZ55JFHrM6dOzs+h4eHW1OmTLEsy7K+/vpry8vLy0pOTnZsj4+PtyRZn332mWVZlnXgwAFLkvXee+859tmxY4clydq5c2eOfw9Afle3bl1r6tSplmVZ1pUrV6yiRYtaa9assSzLskaNGmVVq1Yt0zE3/p38M5UqVbJmzJhhWZZlJSUlWZKs+Pj4m+67Zs0ax9/jqlWrWg899JBlt9tv65rgfhh5QZ5p2LChEhMTlZiYqB9++EHNmzdXy5YtdejQIW3dulV79+5VYGCgAgICFBAQoMKFC+vSpUvat2+fo49KlSrJ09PT8TksLEwnTpy46fmSkpJUqlQplShRwtH2r3/966b7Vq1a1alPSbfsF8DNJSUl6YcfflDXrl0lSV5eXurcufMtb+X8mbS0NA0ZMkQVK1ZUcHCwAgICtHPnTsfIS2Jiojw9PVW/fv0/7adp06aKjIzUwoUL5e3tnf2LglvycnUB+Ofw9/dXZGSk4/N7772nQoUK6d1331VaWppq1qyp+fPnZzquWLFijv8uUKCA0zabzaaMjIy/XduN/dpsNknKkX6Bf5LY2FhdvXpVJUuWdLRZliUfHx+9+eab2epryJAhio+P18SJExUZGSk/Pz89/PDDjtvIfn5+WeqndevWWrx4sX755RdVqVIlWzXAfRFe4DI2m00eHh66ePGiatSooYULFyo0NFRBQUE50n9UVJSOHDmi3377TcWLF5ckx/10ADnr6tWr+uCDDzRp0iQ1a9bMaVv79u21YMECeXt7Kz09PdOxBQoUyNS+YcMG9ezZUx06dJB0bSTmxgn8VapUUUZGhr799ls1adLklnW99tprCggIUOPGjbV27VpFR0f/jauEu+C2EfKM3W7X8ePHdfz4ce3cuVNPP/200tLS1LZtW3Xv3l1FixZVu3bttG7dOh04cEBr167VwIEDdfTo0ds6X9OmTVW2bFnFxMRo27Zt2rBhg0aOHCnp/0ZXAOSMZcuWKTU1VX369FHlypWdfh566CHFxsYqIiJCBw4cUGJiok6dOiW73S7p2irBVatW6fjx40pNTZUklStXTkuWLFFiYqK2bt2qbt26OY2GRkREKCYmRr1791ZcXJzjd8Ynn3ySqbaJEyeqe/fuatSokXbt2pU3XwhyFeEFeeabb75RWFiYwsLCVLt2bW3evFmffvqpGjRooIIFCyohIUF33XWXOnbsqIoVK6pPnz66dOnSbY/EeHp6Ki4uTmlpaapVq5b69u3rWG3k6+ubk5cG/OPFxsaqSZMmKlSoUKZtDz30kH788UdVqlRJLVq0UMOGDVWsWDEtWLBAkjRp0iTFx8erVKlSuvvuuyVJkydPVkhIiOrWrau2bduqefPmqlGjhlO/b7/9th5++GE9+eSTqlChgvr166fz58/ftL4pU6aoU6dOatSokXbv3p3DV4+8ZrMsy3J1EUBe2bBhg+677z7t3btXZcuWdXU5AIDbQHhBvvbZZ58pICBA5cqV0969ezVo0CCFhIRo/fr1ri4NAHCbmLCLfO3cuXMaNmyYDh8+rKJFi6pJkyaaNGmSq8sCAPwNjLwAAACjMGEXAAAYhfACAACMQngBAABGIbwAAACjEF4AuK2ePXuqffv2js8NGjTQM888k+d1rF27VjabTadPn87zcwPIjPACINt69uwpm80mm80mb29vRUZGauzYsbp69WqunnfJkiV65ZVXsrQvgQPIv3jOC4Db0qJFC82ePVt2u11fffWVnnrqKRUoUEDDhw932u/y5cvy9vbOkXMWLlw4R/oBYDZGXgDcFh8fH5UoUULh4eEaMGCAmjRpoqVLlzpu9bz66qsqWbKkoqKiJElHjhxRp06dFBwcrMKFC6tdu3ZObwlOT0/Xc889p+DgYBUpUkTPP/+8/vgYqj/eNrLb7Ro2bJhKlSolHx8fRUZGKjY2VgcPHlTDhg0lSSEhIbLZbOrZs6ckKSMjQ+PHj1fp0qXl5+enatWqadGiRU7n+eqrr1S+fHn5+fmpYcOGTnUCcD3CC4Ac4efnp8uXL0uSVq1apaSkJMXHx2vZsmW6cuWKmjdvrsDAQK1bt04bNmxQQECAWrRo4Thm0qRJmjNnjt5//32tX79ev//+uz777LM/PWePHj20YMECTZ8+XTt37tQ777yjgIAAlSpVSosXL5YkJSUlKTk5WdOmTZMkjR8/Xh988IFmzpypHTt26Nlnn9Wjjz6qb7/9VtK1kNWxY0e1bdtWiYmJ6tu3r1544YXc+toA3A4LALIpJibGateunWVZlpWRkWHFx8dbPj4+1pAhQ6yYmBirePHilt1ud+w/b948KyoqysrIyHC02e12y8/Pz1q+fLllWZYVFhZmTZgwwbH9ypUr1p133uk4j2VZVv369a1BgwZZlmVZSUlJliQrPj7+pjWuWbPGkmSlpqY62i5dumQVLFjQ+u6775z27dOnj9W1a1fLsixr+PDhVnR0tNP2YcOGZeoLgOsw5wXAbVm2bJkCAgJ05coVZWRkqFu3bho9erSeeuopValSxWmey9atW7V3714FBgY69XHp0iXt27dPZ86cUXJysmrXru3Y5uXlpXvuuSfTraPrEhMT5enpqfr162e55r179+rChQtq2rSpU/vly5d19913S5J27tzpVIck3XvvvVk+B4DcR3gBcFsaNmyot99+W97e3ipZsqS8vP7v14m/v7/TvmlpaapZs6bmz5+fqZ9ixYrd1vn9/PyyfUxaWpok6csvv9Qdd9zhtM3Hx+e26gCQ9wgvAG6Lv7+/IiMjs7RvjRo1tHDhQoWGhiooKOim+4SFhen777/XAw88IEm6evWqtmzZoho1atx0/ypVqigjI0PffvutmjRpkmn79ZGf9PR0R1t0dLR8fHx0+PDhW47YVKxYUUuXLnVq27Rp019fJIA8w4RdALmue/fuKlq0qNq1a6d169bpwIEDWrt2rQYOHKijR49KkgYNGqTXXntNcXFx2rVrl5588sk/fUZLRESEYmJi1Lt3b8XFxTn6/OSTTyRJ4eHhstlsWrZsmU6ePKm0tDQFBgZqyJAhevbZZzV37lzt27dP//vf/zRjxgzNnTtXkvTEE09oz549Gjp0qJKSkvTRRx9pzpw5uf0VAcgGwguAXFewYEElJCTorrvuUseOHVWxYkX16dNHly5dcozEDB48WI899phiYmJ07733KjAwUB06dPjTft9++209/PDDevLJJ1WhQgX169dP58+flyTdcccdGjNmjF544QUVL15c//73vyVJr7zyil566SWNHz9eFStWVIsWLfTll1+qdOnSkqS77rpLixcvVlxcnKpVq6aZM2dq3LhxufjtAMgum3Wr2XAAAABuiJEXAABgFMILAAAwCuEFAAAYhfACAACMQngBAABGIbwAAACjEF4AAIBRCC8AAMAohBcAAGAUwgsAADAK4QUAABiF8AIAAIzy/wBkMOKYcoHcmQAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Create confusion matrix\n",
        "conf_matrix = confusion_matrix(y_true, y_pred)\n",
        "\n",
        "# Show confusion matrix\n",
        "ax = plt.subplot()\n",
        "sns.heatmap(conf_matrix, annot=True, ax = ax, cmap='Blues', fmt='g', cbar=False)\n",
        "\n",
        "# Add labels, title and ticks\n",
        "ax.set_xlabel('Predicted')\n",
        "ax.set_ylabel('Actual')\n",
        "ax.set_title('Confusion Matrix')\n",
        "ax.xaxis.set_ticklabels(['Benign', 'Attack'])\n",
        "ax.yaxis.set_ticklabels(['Benign', 'Attack'])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "-H7rWFguYTNZ"
      },
      "source": [
        "Now we can calculate overall accuracy and per class accuracy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uHTInLt2YTNZ",
        "outputId": "238ccc0c-2bbc-4d3c-98ef-2f7101cb1325"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 0.819\n",
            "Precision: 0.000\n",
            "Recall: 0.000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/sklearn/metrics/_classification.py:1469: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ]
        }
      ],
      "source": [
        "# Calculate accuracy\n",
        "acc = accuracy_score(y_true, y_pred, normalize=True, sample_weight=None)\n",
        "precision = precision_score(y_true, y_pred)\n",
        "recall = recall_score(y_true, y_pred)\n",
        "\n",
        "print(f\"Accuracy: {acc:.3f}\")\n",
        "print(f\"Precision: {precision:.3f}\")\n",
        "print(f\"Recall: {recall:.3f}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "ZNzyqAH9YTNZ",
        "outputId": "56e19840-fb6b-4296-96c9-3417c14ebe4b"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-44b9a897-7181-45c9-99be-f65ff3041806\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>type</th>\n",
              "      <th>accuracy</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Benign</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Attack</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-44b9a897-7181-45c9-99be-f65ff3041806')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-bf7edec0-47a9-46ce-9a5f-d846efa46185\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-bf7edec0-47a9-46ce-9a5f-d846efa46185')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-bf7edec0-47a9-46ce-9a5f-d846efa46185 button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-44b9a897-7181-45c9-99be-f65ff3041806 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-44b9a897-7181-45c9-99be-f65ff3041806');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "     type  accuracy\n",
              "0  Benign       1.0\n",
              "1  Attack       0.0"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Calculate per class accuracy\n",
        "cmd = confusion_matrix(y_true, y_pred, normalize=\"true\").diagonal()\n",
        "per_class_accuracy_df = pd.DataFrame([(index, round(value,4)) for index, value in zip(['Benign', 'Attack'], cmd)], columns = ['type', 'accuracy'])\n",
        "per_class_accuracy_df = per_class_accuracy_df.round(2)\n",
        "display(per_class_accuracy_df)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Gfy_LW5zXIj6"
      },
      "source": [
        "We got great results using Pinecone! In [our longer example](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/security/it-threat-detection.ipynb) we also see what happens if we skip the similarity search step and predict values from the model directly. In short, it performs much worse! By adding Pinecone's similarity search over the same model's embeddings we improve threat detection (i.e., \"Attack\") accuracy by over 50%!"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "OiT0JgQxktOC"
      },
      "source": [
        "### Result summary"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "UBBv6-dnfEeX"
      },
      "source": [
        "Using standard vector embeddings with Pinecone's similarity search service, we detected 85% of the attacks while keeping a low 3% false-positive rate. In [our longer example](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/security/it-threat-detection.ipynb), we also showed that our similarity search approach outperforms the direct classification approach that utilizes the classifier's embedding model. Similarity search-based detection gained 50% higher accuracy compared to the direct detector.\n",
        "\n",
        "[Original published results](https://github.com/rambasnet/DeepLearning-IDS/blob/master/graphics/confusion_matrices/) for 02-22-2018 show that the model was able to correctly detect 208520 benign cases out of 208520 benign cases, and 24 (18+1+5) attacks out of 70 attacks in the test set making this model **34.3% accurate in predicting attacks**. For testing purposes, 20% of the data for 02-22-2018 was used.\n",
        "\n",
        "![02-22-2018--6-15%281%29.png](https://raw.githubusercontent.com/rambasnet/DeepLearning-IDS/master/graphics/confusion_matrices/02-22-2018--6-15(1).png)\n",
        "\n",
        "As you can see, the model's performance for creating embeddings for Pinecone was much higher.\n",
        "\n",
        "The model we have created follows the academic paper ([model for the same date](https://github.com/rambasnet/DeepLearning-IDS/blob/master/keras_tensorflow_models/) (02-23-2018)) and is slightly modified, but still a straightforward, sequential, shallow model. We have changed the number of classes from four (Benign, BruteForce-Web, BruteForce-XSS, SQL-Injection) to two (Benign and Attack), only interested in whether we are detecting an attack or not. We have also changed validation metrics to precision and recall. These changes improved our results. Yet, there is still room for further improvements, for example, by adding more data covering multiple days and different types of attacks."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "JdZcEl1pfEeX"
      },
      "source": [
        "## Delete the Index"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "hyxLeOnSfEeX"
      },
      "source": [
        "Delete the index once you are sure that you do not want to use it anymore. Once it is deleted, you cannot reuse it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ywvaJrVkfEeX"
      },
      "outputs": [],
      "source": [
        "pc.delete_index(index_name)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Hu2mIbHms3k7"
      },
      "source": [
        "---"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "environment": {
      "name": "tf2-gpu.2-3.m65",
      "type": "gcloud",
      "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-3:m65"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.16"
    },
    "vscode": {
      "interpreter": {
        "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}