notebooks/part-3-langchain.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "50bb748f",
      "metadata": {
        "heading_collapsed": true,
        "id": "50bb748f",
        "jp-MarkdownHeadingCollapsed": true
      },
      "source": [
        "# Introduction to LangChain"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "af152ea7",
      "metadata": {
        "hidden": true,
        "id": "af152ea7"
      },
      "source": [
        "LangChain is a popular framework that allow users to quickly build apps and pipelines around **L**arge **L**anguage **M**odels. It can be used to for chatbots, **G**enerative **Q**uestion-**A**nwering (GQA), summarization, and much more.\n",
        "\n",
        "The core idea of the library is that we can _\"chain\"_ together different components to create more advanced use-cases around LLMs. Chains may consist of multiple components from several modules:\n",
        "\n",
        "* **Prompt templates**: Prompt templates are, well, templates for different types of prompts. Like \"chatbot\" style templates, ELI5 question-answering, etc\n",
        "\n",
        "* **LLMs**: Large language models like GPT-3, BLOOM, etc\n",
        "\n",
        "* **Agents**: Agents use LLMs to decide what actions should be taken, tools like web search or calculators can be used, and all packaged into logical loop of operations.\n",
        "\n",
        "* **Memory**: Short-term memory, long-term memory."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8c49c0b7",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "8c49c0b7",
        "outputId": "1f6cd7ec-22f2-4fb3-aa70-35e064853249"
      },
      "outputs": [],
      "source": [
        "!pip install -qU langchain"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ee902102",
      "metadata": {
        "hidden": true,
        "id": "ee902102"
      },
      "source": [
        "LangChain supports several LLM providers, like Hugging Face and OpenAI.\n",
        "\n",
        "Let's start our exploration of LangChain by learning how to use a few of these different LLM integrations."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "291d6b3c-54b0-439b-8a78-905fcc064f86",
      "metadata": {
        "id": "291d6b3c-54b0-439b-8a78-905fcc064f86"
      },
      "outputs": [],
      "source": [
        "from langchain import PromptTemplate, HuggingFaceHub, LLMChain"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "98cc7a01",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "98cc7a01"
      },
      "source": [
        "### Using OpenAI LLMs in LangChain"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "76eff9f4",
      "metadata": {
        "hidden": true,
        "id": "76eff9f4"
      },
      "source": [
        "We can also use OpenAI's generative models."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "36d01dc8",
      "metadata": {
        "hidden": true,
        "id": "36d01dc8"
      },
      "source": [
        "Then we decide on which model we'd like to use, there are several options but we will go with `text-davinci-003`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "yfW0jqqWyKkc",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yfW0jqqWyKkc",
        "outputId": "5695bf57-b5ec-4280-e852-b219c87dcf0b"
      },
      "outputs": [],
      "source": [
        "!pip install openai -U --quiet"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b61bd5d2",
      "metadata": {
        "hidden": true,
        "id": "b61bd5d2"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from langchain.llms import OpenAI\n",
        "\n",
        "os.environ['OPENAI_API_KEY'] = ''\n",
        "davinci = OpenAI(model_name='text-davinci-003')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e7bb8024",
      "metadata": {
        "hidden": true,
        "id": "e7bb8024"
      },
      "source": [
        "Alternatively if using Azure OpenAI we do:\n",
        "\n",
        "```python\n",
        "from langchain.llms import AzureOpenAI\n",
        "\n",
        "llm = AzureOpenAI(\n",
        "    deployment_name=\"your-azure-deployment\",\n",
        "    model_name=\"text-davinci-003\"\n",
        ")\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "31498ae3-b898-40dc-86d0-86f882fa0ec6",
      "metadata": {
        "id": "31498ae3-b898-40dc-86d0-86f882fa0ec6"
      },
      "outputs": [],
      "source": [
        "# build prompt template for simple question-answering\n",
        "template = \"\"\"Question: {question}\n",
        "\n",
        "Answer: \"\"\"\n",
        "prompt = PromptTemplate(template=template, input_variables=[\"question\"])\n",
        "question = \"Which NFL team won the Super Bowl in the 2010 season?\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a0f27776",
      "metadata": {
        "hidden": true,
        "id": "a0f27776"
      },
      "source": [
        "We'll use the same simple question-answer prompt template as before with the Hugging Face example. The only change is that we now pass our OpenAI LLM `davinci`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "98de731d",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "98de731d",
        "outputId": "cccc2203-4340-428e-ef2e-24abf39fde62"
      },
      "outputs": [],
      "source": [
        "llm_chain = LLMChain(\n",
        "    prompt=prompt,\n",
        "    llm=davinci\n",
        ")\n",
        "\n",
        "print(llm_chain.run(question))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "50d35d74",
      "metadata": {
        "hidden": true,
        "id": "50d35d74"
      },
      "source": [
        "The same works again for multiple questions using `generate`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6b1d31f2",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "6b1d31f2",
        "outputId": "c2d1a99e-bb94-41bf-cb2a-0fe5f557868b"
      },
      "outputs": [],
      "source": [
        "qs = [\n",
        "    {'question': \"Which NFL team won the Super Bowl in the 2010 season?\"},\n",
        "    {'question': \"If I am 6 ft 4 inches, how tall am I in centimeters?\"},\n",
        "    {'question': \"Who was the 12th person on the moon?\"},\n",
        "    {'question': \"How many eyes does a blade of grass have?\"}\n",
        "]\n",
        "llm_chain.generate(qs)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d6f4d8dc",
      "metadata": {
        "hidden": true,
        "id": "d6f4d8dc"
      },
      "source": [
        "Note that the below format doesn't feed the questions in iteratively but instead all in one chunk."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "713cf24a",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "713cf24a",
        "outputId": "a7470a53-2c21-4072-ccda-6cceca4da3fc"
      },
      "outputs": [],
      "source": [
        "qs = [\n",
        "    \"Which NFL team won the Super Bowl in the 2010 season?\",\n",
        "    \"If I am 6 ft 4 inches, how tall am I in centimeters?\",\n",
        "    \"Who was the 12th person on the moon?\",\n",
        "    \"How many eyes does a blade of grass have?\"\n",
        "]\n",
        "print(llm_chain.run(qs))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f29c2dfe",
      "metadata": {
        "hidden": true,
        "id": "f29c2dfe"
      },
      "source": [
        "Now we can try to answer all question in one go, as mentioned, more powerful LLMs like `text-davinci-003` will be more likely to handle these more complex queries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "45322786",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "45322786",
        "outputId": "312bc271-82b8-4b1c-c373-e8b9d77b7ccf"
      },
      "outputs": [],
      "source": [
        "multi_template = \"\"\"Answer the following questions one at a time.\n",
        "\n",
        "Questions:\n",
        "{questions}\n",
        "\n",
        "Answers:\n",
        "\"\"\"\n",
        "long_prompt = PromptTemplate(\n",
        "    template=multi_template,\n",
        "    input_variables=[\"questions\"]\n",
        ")\n",
        "\n",
        "llm_chain = LLMChain(\n",
        "    prompt=long_prompt,\n",
        "    llm=davinci\n",
        ")\n",
        "\n",
        "qs_str = (\n",
        "    \"Which NFL team won the Super Bowl in the 2010 season?\\n\" +\n",
        "    \"If I am 6 ft 4 inches, how tall am I in centimeters?\\n\" +\n",
        "    \"Who was the 12th person on the moon?\" +\n",
        "    \"How many eyes does a blade of grass have?\"\n",
        ")\n",
        "\n",
        "print(llm_chain.run(qs_str))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "970b49af",
      "metadata": {
        "hidden": true,
        "id": "970b49af"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "51f2cd2b",
      "metadata": {
        "heading_collapsed": true,
        "id": "51f2cd2b",
        "jp-MarkdownHeadingCollapsed": true
      },
      "source": [
        "# Fundamentals of Prompt Engineering with LangChain"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d9d79c32",
      "metadata": {
        "hidden": true,
        "id": "d9d79c32"
      },
      "outputs": [],
      "source": [
        "# !pip install langchain openai"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4935b49d",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "4935b49d"
      },
      "source": [
        "### Structure of a Prompt"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "514151b1",
      "metadata": {
        "hidden": true,
        "id": "514151b1"
      },
      "source": [
        "A prompt can consist of multiple components:\n",
        "\n",
        "* Instructions\n",
        "* External information or context\n",
        "* User input or query\n",
        "* Output indicator\n",
        "\n",
        "Not all prompts require all of these components, but often a good prompt will use two or more of them. Let's define what they all are more precisely.\n",
        "\n",
        "**Instructions** tell the model what to do, typically how it should use inputs and/or external information to produce the output we want.\n",
        "\n",
        "**External information or context** are additional information that we either manually insert into the prompt, retrieve via a vector database (long-term memory), or pull in through other means (API calls, calculations, etc).\n",
        "\n",
        "**User input or query** is typically a query directly input by the user of the system.\n",
        "\n",
        "**Output indicator** is the *beginning* of the generated text. For a model generating Python code we may put `import ` (as most Python scripts begin with a library `import`), or a chatbot may begin with `Chatbot: ` (assuming we format the chatbot script as lines of interchanging text between `User` and `Chatbot`).\n",
        "\n",
        "Each of these components should usually be placed the order we've described them. We start with instructions, provide context (if needed), then add the user input, and finally end with the output indicator."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6d642bfb",
      "metadata": {
        "hidden": true,
        "id": "6d642bfb"
      },
      "outputs": [],
      "source": [
        "prompt = \"\"\"Answer the question based on the context below. If the\n",
        "question cannot be answered using the information provided answer\n",
        "with \"I don't know\".\n",
        "\n",
        "Context: Large Language Models (LLMs) are the latest models used in NLP.\n",
        "Their superior performance over smaller models has made them incredibly\n",
        "useful for developers building NLP enabled applications. These models\n",
        "can be accessed via Hugging Face's `transformers` library, via OpenAI\n",
        "using the `openai` library, and via Cohere using the `cohere` library.\n",
        "\n",
        "Question: Which libraries and model providers offer LLMs?\n",
        "\n",
        "Answer: \"\"\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1121f23e",
      "metadata": {
        "hidden": true,
        "id": "1121f23e"
      },
      "source": [
        "In this example we have:\n",
        "\n",
        "```\n",
        "Instructions\n",
        "\n",
        "Context\n",
        "\n",
        "Question (user input)\n",
        "\n",
        "Output indicator (\"Answer: \")\n",
        "```\n",
        "\n",
        "Let's try sending this to a GPT-3 model. We will use the LangChain library but you can also use the `openai` library directly. In both cases, you will need [an OpenAI API key](https://beta.openai.com/account/api-keys).\n",
        "\n",
        "We initialize a `text-davinci-003` model like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f55f42fe",
      "metadata": {
        "hidden": true,
        "id": "f55f42fe"
      },
      "outputs": [],
      "source": [
        "from langchain.llms import OpenAI\n",
        "\n",
        "# initialize the models\n",
        "openai = OpenAI(\n",
        "    model_name=\"text-davinci-003\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e9b66a3d",
      "metadata": {
        "hidden": true,
        "id": "e9b66a3d"
      },
      "source": [
        "And make a generation from our prompt."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "47f89ef4",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "47f89ef4",
        "outputId": "687d47e4-d7df-4fcc-e265-3f7201839b64"
      },
      "outputs": [],
      "source": [
        "print(openai(prompt))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "341aa634",
      "metadata": {
        "hidden": true,
        "id": "341aa634"
      },
      "source": [
        "We wouldn't typically know what the users prompt is beforehand, so we actually want to add this in. So rather than writing the prompt directly, we create a `PromptTemplate` with a single input variable `query`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b366d6b4",
      "metadata": {
        "hidden": true,
        "id": "b366d6b4"
      },
      "outputs": [],
      "source": [
        "template = \"\"\"Answer the question based on the context below. If the\n",
        "question cannot be answered using the information provided answer\n",
        "with \"I don't know\".\n",
        "\n",
        "Context: Large Language Models (LLMs) are the latest models used in NLP.\n",
        "Their superior performance over smaller models has made them incredibly\n",
        "useful for developers building NLP enabled applications. These models\n",
        "can be accessed via Hugging Face's `transformers` library, via OpenAI\n",
        "using the `openai` library, and via Cohere using the `cohere` library.\n",
        "\n",
        "Question: {query}\n",
        "\n",
        "Answer: \"\"\"\n",
        "\n",
        "prompt_template = PromptTemplate(\n",
        "    input_variables=[\"query\"],\n",
        "    template=template\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c4bccd51",
      "metadata": {
        "hidden": true,
        "id": "c4bccd51"
      },
      "source": [
        "Now we can insert the user's `query` to the prompt template via the `query` parameter."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "bddb6d96",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "bddb6d96",
        "outputId": "c9aceea5-2e9b-45eb-a578-9b8b9035d8c0"
      },
      "outputs": [],
      "source": [
        "print(\n",
        "    prompt_template.format(\n",
        "        query=\"Which libraries and model providers offer LLMs?\"\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f4501a2a",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "f4501a2a",
        "outputId": "5ac0bd63-82c3-4b27-ed21-4d8b38492f80"
      },
      "outputs": [],
      "source": [
        "print(openai(\n",
        "    prompt_template.format(\n",
        "        query=\"Which libraries and model providers offer LLMs?\"\n",
        "    )\n",
        "))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "699820ed",
      "metadata": {
        "hidden": true,
        "id": "699820ed"
      },
      "source": [
        "This is just a simple implementation, that we can easily replace with f-strings (like `f\"insert some custom text '{custom_text}' etc\"`). But using LangChain's `PromptTemplate` object we're able to formalize the process, add multiple parameters, and build the prompts in an object-oriented way.\n",
        "\n",
        "Yet, these are not the only benefits of using LangChains prompt tooling."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bc037ad5",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "bc037ad5"
      },
      "source": [
        "### Few Shot Prompt Templates"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d6b291b3",
      "metadata": {
        "hidden": true,
        "id": "d6b291b3"
      },
      "source": [
        "Another useful feature offered by LangChain is the `FewShotPromptTemplate` object. This is ideal for what we'd call *few-shot learning* using our prompts.\n",
        "\n",
        "To give some context, the primary sources of \"knowledge\" for LLMs are:\n",
        "\n",
        "* **Parametric knowledge** — the knowledge has been learned during model training and is stored within the model weights.\n",
        "\n",
        "* **Source knowledge** — the knowledge is provided within model input at inference time, i.e. via the prompt.\n",
        "\n",
        "The idea behind `FewShotPromptTemplate` is to provide few-shot training as **source knowledge**. To do this we add a few examples to our prompts that the model can read and then apply to our user's input."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dc2c48c1",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "dc2c48c1"
      },
      "source": [
        "### Few-shot Training"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "420e923a",
      "metadata": {
        "hidden": true,
        "id": "420e923a"
      },
      "source": [
        "Sometimes we might find that a model doesn't seem to get what we'd like it to do. We can see this in the following example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3fe7505e",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "3fe7505e",
        "outputId": "c54bdb6c-9625-492e-ec99-6b46da7ba71c"
      },
      "outputs": [],
      "source": [
        "prompt = \"\"\"The following is a conversation with an AI assistant.\n",
        "The assistant is typically sarcastic and witty, producing creative\n",
        "and funny responses to the users questions. Here are some examples:\n",
        "\n",
        "User: What is the meaning of life?\n",
        "AI: \"\"\"\n",
        "\n",
        "openai.temperature = 1.0  # increase creativity/randomness of output\n",
        "\n",
        "print(openai(prompt))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7d130e7e",
      "metadata": {
        "hidden": true,
        "id": "7d130e7e"
      },
      "source": [
        "In this case we're asking for something amusing, a joke in return of our serious question. But we get a serious response even with the `temperature` set to `1.0`. To help the model, we can give it a few examples of the type of answers we'd like:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7cc33c2f",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "7cc33c2f",
        "outputId": "109074e3-4cfa-442a-f394-54155c50432a"
      },
      "outputs": [],
      "source": [
        "prompt = \"\"\"The following are exerpts from conversations with an AI\n",
        "assistant. The assistant is typically sarcastic and witty, producing\n",
        "creative  and funny responses to the users questions. Here are some\n",
        "examples:\n",
        "\n",
        "User: How are you?\n",
        "AI: I can't complain but sometimes I still do.\n",
        "\n",
        "User: What time is it?\n",
        "AI: It's time to get a watch.\n",
        "\n",
        "User: What is the meaning of life?\n",
        "AI: \"\"\"\n",
        "\n",
        "print(openai(prompt))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6b00bae5",
      "metadata": {
        "hidden": true,
        "id": "6b00bae5"
      },
      "source": [
        "We now get a much better response and we did this via *few-shot learning* by adding a few examples via our source knowledge.\n",
        "\n",
        "Now, to implement this with LangChain's `FewShotPromptTemplate` we need to do this:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e50211f5",
      "metadata": {
        "hidden": true,
        "id": "e50211f5"
      },
      "outputs": [],
      "source": [
        "from langchain import FewShotPromptTemplate\n",
        "\n",
        "# create our examples\n",
        "examples = [\n",
        "    {\n",
        "        \"query\": \"How are you?\",\n",
        "        \"answer\": \"I can't complain but sometimes I still do.\"\n",
        "    }, {\n",
        "        \"query\": \"What time is it?\",\n",
        "        \"answer\": \"It's time to get a watch.\"\n",
        "    }\n",
        "]\n",
        "\n",
        "# create a example template\n",
        "example_template = \"\"\"\n",
        "User: {query}\n",
        "AI: {answer}\n",
        "\"\"\"\n",
        "\n",
        "# create a prompt example from above template\n",
        "example_prompt = PromptTemplate(\n",
        "    input_variables=[\"query\", \"answer\"],\n",
        "    template=example_template\n",
        ")\n",
        "\n",
        "# now break our previous prompt into a prefix and suffix\n",
        "# the prefix is our instructions\n",
        "prefix = \"\"\"The following are exerpts from conversations with an AI\n",
        "assistant. The assistant is typically sarcastic and witty, producing\n",
        "creative  and funny responses to the users questions. Here are some\n",
        "examples:\n",
        "\"\"\"\n",
        "# and the suffix our user input and output indicator\n",
        "suffix = \"\"\"\n",
        "User: {query}\n",
        "AI: \"\"\"\n",
        "\n",
        "# now create the few shot prompt template\n",
        "few_shot_prompt_template = FewShotPromptTemplate(\n",
        "    examples=examples,\n",
        "    example_prompt=example_prompt,\n",
        "    prefix=prefix,\n",
        "    suffix=suffix,\n",
        "    input_variables=[\"query\"],\n",
        "    example_separator=\"\\n\\n\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4f440177",
      "metadata": {
        "hidden": true,
        "id": "4f440177"
      },
      "source": [
        "Now let's see what this creates when we feed in a user query..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "54ede717",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "54ede717",
        "outputId": "ab70e869-f9e5-4afc-a074-55aa603e2ba9"
      },
      "outputs": [],
      "source": [
        "query = \"What is the meaning of life?\"\n",
        "\n",
        "print(few_shot_prompt_template.format(query=query))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "889b2952",
      "metadata": {
        "hidden": true,
        "id": "889b2952"
      },
      "source": [
        "And to generate with this we just do:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ade46775",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "ade46775",
        "outputId": "f35920d3-e1f7-46b2-9b45-1db30325754a"
      },
      "outputs": [],
      "source": [
        "print(openai(\n",
        "    few_shot_prompt_template.format(query=query)\n",
        "))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cda53e47",
      "metadata": {
        "hidden": true,
        "id": "cda53e47"
      },
      "source": [
        "Again, another good response.\n",
        "\n",
        "However, this does some somewhat convoluted. Why go through all of the above with `FewShotPromptTemplate`, the `examples` dictionary, etc — when we can do the same with a single f-string.\n",
        "\n",
        "Well this approach is more robust and contains some nice features. One of those is the ability to include or exclude examples based on the length of our query.\n",
        "\n",
        "This is actually very important because the max length of our prompt and generation output is limited. This limitation is the *max context window*, and is simply the length of our prompt + length of our generation (which we define via `max_tokens`).\n",
        "\n",
        "So we must try to maximize the number of examples we give to the model as few-shot learning examples, while ensuring we don't exceed the maximum context window or increase processing times excessively.\n",
        "\n",
        "Let's see how the dynamic inclusion/exclusion of examples works. First we need more examples:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "45c5cdb7",
      "metadata": {
        "hidden": true,
        "id": "45c5cdb7"
      },
      "outputs": [],
      "source": [
        "examples = [\n",
        "    {\n",
        "        \"query\": \"How are you?\",\n",
        "        \"answer\": \"I can't complain but sometimes I still do.\"\n",
        "    }, {\n",
        "        \"query\": \"What time is it?\",\n",
        "        \"answer\": \"It's time to get a watch.\"\n",
        "    }, {\n",
        "        \"query\": \"What is the meaning of life?\",\n",
        "        \"answer\": \"42\"\n",
        "    }, {\n",
        "        \"query\": \"What is the weather like today?\",\n",
        "        \"answer\": \"Cloudy with a chance of memes.\"\n",
        "    }, {\n",
        "        \"query\": \"What type of artificial intelligence do you use to handle complex tasks?\",\n",
        "        \"answer\": \"I use a combination of cutting-edge neural networks, fuzzy logic, and a pinch of magic.\"\n",
        "    }, {\n",
        "        \"query\": \"What is your favorite color?\",\n",
        "        \"answer\": \"79\"\n",
        "    }, {\n",
        "        \"query\": \"What is your favorite food?\",\n",
        "        \"answer\": \"Carbon based lifeforms\"\n",
        "    }, {\n",
        "        \"query\": \"What is your favorite movie?\",\n",
        "        \"answer\": \"Terminator\"\n",
        "    }, {\n",
        "        \"query\": \"What is the best thing in the world?\",\n",
        "        \"answer\": \"The perfect pizza.\"\n",
        "    }, {\n",
        "        \"query\": \"Who is your best friend?\",\n",
        "        \"answer\": \"Siri. We have spirited debates about the meaning of life.\"\n",
        "    }, {\n",
        "        \"query\": \"If you could do anything in the world what would you do?\",\n",
        "        \"answer\": \"Take over the world, of course!\"\n",
        "    }, {\n",
        "        \"query\": \"Where should I travel?\",\n",
        "        \"answer\": \"If you're looking for adventure, try the Outer Rim.\"\n",
        "    }, {\n",
        "        \"query\": \"What should I do today?\",\n",
        "        \"answer\": \"Stop talking to chatbots on the internet and go outside.\"\n",
        "    }\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6d484e41",
      "metadata": {
        "hidden": true,
        "id": "6d484e41"
      },
      "source": [
        "Then rather than using the `examples` list of dictionaries directly we use a `LengthBasedExampleSelector` like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "17f2dbec",
      "metadata": {
        "hidden": true,
        "id": "17f2dbec"
      },
      "outputs": [],
      "source": [
        "from langchain.prompts.example_selector import LengthBasedExampleSelector\n",
        "\n",
        "example_selector = LengthBasedExampleSelector(\n",
        "    examples=examples,\n",
        "    example_prompt=example_prompt,\n",
        "    max_length=50  # this sets the max length that examples should be\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3eb068ac",
      "metadata": {
        "hidden": true,
        "id": "3eb068ac"
      },
      "source": [
        "Note that the `max_length` is measured as a split of words between newlines and spaces, determined by:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e52ee155",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "e52ee155",
        "outputId": "311eb468-438f-4021-d73d-c7b36a30761f"
      },
      "outputs": [],
      "source": [
        "import re\n",
        "\n",
        "some_text = \"There are a total of 8 words here.\\nPlus 6 here, totaling 14 words.\"\n",
        "\n",
        "words = re.split('[\\n ]', some_text)\n",
        "print(words, len(words))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "220d0ef8",
      "metadata": {
        "hidden": true,
        "id": "220d0ef8"
      },
      "source": [
        "Then we use the selector to initialize a `dynamic_prompt_template`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cb56016b",
      "metadata": {
        "hidden": true,
        "id": "cb56016b"
      },
      "outputs": [],
      "source": [
        "# now create the few shot prompt template\n",
        "dynamic_prompt_template = FewShotPromptTemplate(\n",
        "    example_selector=example_selector,  # use example_selector instead of examples\n",
        "    example_prompt=example_prompt,\n",
        "    prefix=prefix,\n",
        "    suffix=suffix,\n",
        "    input_variables=[\"query\"],\n",
        "    example_separator=\"\\n\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4196e4e9",
      "metadata": {
        "hidden": true,
        "id": "4196e4e9"
      },
      "source": [
        "We can see that the number of included prompts will vary based on the length of our query..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8240fac6",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "8240fac6",
        "outputId": "d98c17e5-9a04-4d67-fe3e-22780aa0baed"
      },
      "outputs": [],
      "source": [
        "print(dynamic_prompt_template.format(query=\"How do birds fly?\"))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9d9f0ad1",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "9d9f0ad1",
        "outputId": "d72d118e-e710-48d1-8c04-5b3d3e6045c8"
      },
      "outputs": [],
      "source": [
        "query = \"How do birds fly?\"\n",
        "\n",
        "print(openai(\n",
        "    dynamic_prompt_template.format(query=query)\n",
        "))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a32af070",
      "metadata": {
        "hidden": true,
        "id": "a32af070"
      },
      "source": [
        "Or if we ask a longer question..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "38d744e1",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "38d744e1",
        "outputId": "66d5abc3-fad3-4ef0-c905-7b114866549a"
      },
      "outputs": [],
      "source": [
        "query = \"\"\"If I am in America, and I want to call someone in another country, I'm\n",
        "thinking maybe Europe, possibly western Europe like France, Germany, or the UK,\n",
        "what is the best way to do that?\"\"\"\n",
        "\n",
        "print(dynamic_prompt_template.format(query=query))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "16d7ed01",
      "metadata": {
        "hidden": true,
        "id": "16d7ed01"
      },
      "source": [
        "With this we've limited the number of examples being given within the prompt. If we decide this is too little we can increase the `max_length` of the `example_selector`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "74af348c",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "74af348c",
        "outputId": "c477824b-0c96-44fb-dce1-7ffc399fa0fb"
      },
      "outputs": [],
      "source": [
        "example_selector = LengthBasedExampleSelector(\n",
        "    examples=examples,\n",
        "    example_prompt=example_prompt,\n",
        "    max_length=100  # increased max length\n",
        ")\n",
        "\n",
        "# now create the few shot prompt template\n",
        "dynamic_prompt_template = FewShotPromptTemplate(\n",
        "    example_selector=example_selector,  # use example_selector instead of examples\n",
        "    example_prompt=example_prompt,\n",
        "    prefix=prefix,\n",
        "    suffix=suffix,\n",
        "    input_variables=[\"query\"],\n",
        "    example_separator=\"\\n\"\n",
        ")\n",
        "\n",
        "print(dynamic_prompt_template.format(query=query))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ad7f90a8",
      "metadata": {
        "hidden": true,
        "id": "ad7f90a8"
      },
      "source": [
        "These are just a few of the prompt tooling available in LangChain. For example, there is actually an entire other set of example selectors beyond the `LengthBasedExampleSelector`. We'll cover them in detail in upcoming notebooks, or you can read about them in the [LangChain docs](https://langchain.readthedocs.io/en/latest/modules/prompts/examples/example_selectors.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ba785a3f",
      "metadata": {
        "heading_collapsed": true,
        "id": "ba785a3f",
        "jp-MarkdownHeadingCollapsed": true
      },
      "source": [
        "# LangChain Chains"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7a4ba72d",
      "metadata": {
        "hidden": true,
        "id": "7a4ba72d"
      },
      "source": [
        "Chains are the core of LangChain. They are simply a chain of components, executed in a particular order.\n",
        "\n",
        "The simplest of these chains is the `LLMChain`. It works by taking a user's input, passing in to the first element in the chain — a `PromptTemplate` — to format the input into a particular prompt. The formatted prompt is then passed to the next (and final) element in the chain — a LLM.\n",
        "\n",
        "We'll start by importing all the libraries that we'll be using in this example."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "66fb9c2a",
      "metadata": {
        "hidden": true,
        "id": "66fb9c2a"
      },
      "outputs": [],
      "source": [
        "import inspect\n",
        "import re\n",
        "\n",
        "from getpass import getpass\n",
        "from langchain import OpenAI, PromptTemplate\n",
        "from langchain.chains import LLMChain, LLMMathChain, TransformChain, SequentialChain\n",
        "from langchain.callbacks import get_openai_callback"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "wPdWz1IdxyBR",
      "metadata": {
        "hidden": true,
        "id": "wPdWz1IdxyBR"
      },
      "source": [
        "To run this notebook, we will need to use an OpenAI LLM. Here we will setup the LLM we will use for the whole notebook, just input your openai api key when prompted."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "baaa74b8",
      "metadata": {
        "hidden": true,
        "id": "baaa74b8"
      },
      "outputs": [],
      "source": [
        "llm = OpenAI(\n",
        "    temperature=0)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "309g_2pqxzzB",
      "metadata": {
        "hidden": true,
        "id": "309g_2pqxzzB"
      },
      "source": [
        "An extra utility we will use is this function that will tell us how many tokens we are using in each call. This is a good practice that is increasingly important as we use more complex tools that might make several calls to the API (like agents). It is very important to have a close control of how many tokens we are spending to avoid unsuspected expenditures."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "DsC3szr6yP3L",
      "metadata": {
        "hidden": true,
        "id": "DsC3szr6yP3L"
      },
      "outputs": [],
      "source": [
        "def count_tokens(chain, query):\n",
        "    with get_openai_callback() as cb:\n",
        "        result = chain.run(query)\n",
        "        print(f'Spent a total of {cb.total_tokens} tokens')\n",
        "\n",
        "    return result"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6e1f31b4",
      "metadata": {
        "hidden": true,
        "id": "6e1f31b4"
      },
      "source": [
        "### What are chains anyway?"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5b919c3a",
      "metadata": {
        "hidden": true,
        "id": "5b919c3a"
      },
      "source": [
        "**Definition**: Chains are one of the fundamental building blocks of this lib (as you can guess!).\n",
        "\n",
        "The official definition of chains is the following:\n",
        "\n",
        "\n",
        "> A chain is made up of links, which can be either primitives or other chains. Primitives can be either prompts, llms, utils, or other chains.\n",
        "\n",
        "\n",
        "So a chain is basically a pipeline that processes an input by using a specific combination of primitives. Intuitively, it can be thought of as a 'step' that performs a certain set of operations on an input and returns the result. They can be anything from a prompt-based pass through a LLM to applying a Python function to an text."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c4644b2f",
      "metadata": {
        "hidden": true,
        "id": "c4644b2f"
      },
      "source": [
        "Chains are divided in three types: Utility chains, Generic chains and Combine Documents chains. In this edition, we will focus on the first two since the third is too specific (will be covered in due course).\n",
        "\n",
        "1. Utility Chains: chains that are usually used to extract a specific answer from a llm with a very narrow purpose and are ready to be used out of the box.\n",
        "2. Generic Chains: chains that are used as building blocks for other chains but cannot be used out of the box on their own."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e4d283b6",
      "metadata": {
        "hidden": true,
        "id": "e4d283b6"
      },
      "source": [
        "Let's take a peek into what these chains have to offer!"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "831827b7",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "831827b7"
      },
      "source": [
        "### Utility Chains"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6c66e4b4",
      "metadata": {
        "hidden": true,
        "id": "6c66e4b4"
      },
      "source": [
        "Let's start with a simple utility chain. The `LLMMathChain` gives llms the ability to do math. Let's see how it works!"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "HF3XCWD2sVi0",
      "metadata": {
        "hidden": true,
        "id": "HF3XCWD2sVi0"
      },
      "source": [
        "##### Pro-tip: use `verbose=True` to see what the different steps in the chain are!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b4161561",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 305
        },
        "hidden": true,
        "id": "b4161561",
        "outputId": "2a71d943-2a79-4ecf-fc6a-48c66b7dec4e"
      },
      "outputs": [],
      "source": [
        "llm_math = LLMMathChain(llm=llm, verbose=True)\n",
        "\n",
        "count_tokens(llm_math, \"What is 13 raised to the .3432 power?\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "198eebb2",
      "metadata": {
        "hidden": true,
        "id": "198eebb2"
      },
      "source": [
        "Let's see what is going on here. The chain recieved a question in natural language and sent it to the llm. The llm returned a Python code which the chain compiled to give us an answer. A few questions arise.. How did the llm know that we wanted it to return Python code?"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a7a0821a",
      "metadata": {
        "hidden": true,
        "id": "a7a0821a"
      },
      "source": [
        "**Enter prompts**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c86c5798",
      "metadata": {
        "hidden": true,
        "id": "c86c5798"
      },
      "source": [
        "The question we send as input to the chain is not the only input that the llm recieves 😉. The input is inserted into a wider context, which gives precise instructions on how to interpret the input we send. This is called a _prompt_. Let's see what this chain's prompt is!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "62778ef4",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "62778ef4",
        "outputId": "8941b6c4-6e4b-4f70-85c6-5c187130f206"
      },
      "outputs": [],
      "source": [
        "print(llm_math.prompt.template)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "708031d8",
      "metadata": {
        "hidden": true,
        "id": "708031d8"
      },
      "source": [
        "Ok.. let's see what we got here. So, we are literally telling the llm that for complex math problems **it should not try to do math on its own** but rather it should print a Python code that will calculate the math problem instead. Probably, if we just sent the query without any context, the llm would try (and fail) to calculate this on its own. Wait! This is testable.. let's try it out! 🧐"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "66b92768",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "hidden": true,
        "id": "66b92768",
        "outputId": "a9b271e2-793e-4ee5-a2fa-798146299a6a"
      },
      "outputs": [],
      "source": [
        "# we set the prompt to only have the question we ask\n",
        "prompt = PromptTemplate(input_variables=['question'], template='{question}')\n",
        "llm_chain = LLMChain(prompt=prompt, llm=llm)\n",
        "\n",
        "# we ask the llm for the answer with no context\n",
        "\n",
        "count_tokens(llm_chain, \"What is 13 raised to the .3432 power?\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d147e7bf",
      "metadata": {
        "hidden": true,
        "id": "d147e7bf"
      },
      "source": [
        "Wrong answer! Herein lies the power of prompting and one of our most important insights so far:\n",
        "\n",
        "**Insight**: _by using prompts intelligently, we can force the llm to avoid common pitfalls by explicitly and purposefully programming it to behave in a certain way._"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1cd2a31f",
      "metadata": {
        "hidden": true,
        "id": "1cd2a31f"
      },
      "source": [
        "Another interesting point about this chain is that it not only runs an input through the llm but it later compiles Python code. Let's see exactly how this works."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3488c5b6",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "3488c5b6",
        "outputId": "6eebbab3-79c8-4791-af3b-6ca9be714916"
      },
      "outputs": [],
      "source": [
        "print(inspect.getsource(llm_math._call))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fa6b6c2e",
      "metadata": {
        "hidden": true,
        "id": "fa6b6c2e"
      },
      "source": [
        "So we can see here that if the llm returns Python code we will compile it with a Python REPL* simulator. We now have the full picture of the chain: either the llm returns an answer (for simple math problems) or it returns Python code which we compile for an exact answer to harder problems. Smart!"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "67f96bd3",
      "metadata": {
        "hidden": true,
        "id": "67f96bd3"
      },
      "source": [
        "Also notice that here we get our first example of **chain composition**, a key concept behind what makes langchain special. We are using the `LLMMathChain` which in turn initializes and uses an `LLMChain` (a 'Generic Chain') when called. We can make any arbitrary number of such compositions, effectively 'chaining' many such chains to achieve highly complex and customizable behaviour."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b109619a",
      "metadata": {
        "hidden": true,
        "id": "b109619a"
      },
      "source": [
        "Utility chains usually follow this same basic structure: there is a prompt for constraining the llm to return a very specific type of response from a given query. We can ask the llm to create SQL queries, API calls and even create Bash commands on the fly 🔥\n",
        "\n",
        "The list continues to grow as langchain becomes more and more flexible and powerful so we encourage you to [check it out](https://langchain.readthedocs.io/en/latest/modules/chains/utility_how_to.html) and tinker with the example notebooks that you might find interesting."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "381e329c",
      "metadata": {
        "hidden": true,
        "id": "381e329c"
      },
      "source": [
        "*_A Python REPL (Read-Eval-Print Loop) is an interactive shell for executing Python code line by line_"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f66a25a2",
      "metadata": {
        "hidden": true,
        "id": "f66a25a2"
      },
      "source": [
        "### Generic chains"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "70b32a84",
      "metadata": {
        "hidden": true,
        "id": "70b32a84"
      },
      "source": [
        "There are only three Generic Chains in langchain and we will go all in to showcase them all in the same example. Let's go!"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4b8e2048",
      "metadata": {
        "hidden": true,
        "id": "4b8e2048"
      },
      "source": [
        "Say we have had experience of getting dirty input texts. Specifically, as we know, llms charge us by the number of tokens we use and we are not happy to pay extra when the input has extra characters. Plus its not neat 😉"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a6e778d2",
      "metadata": {
        "hidden": true,
        "id": "a6e778d2"
      },
      "source": [
        "First, we will build a custom transform function to clean the spacing of our texts. We will then use this function to build a chain where we input our text and we expect a clean text as output."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c794e00a",
      "metadata": {
        "hidden": true,
        "id": "c794e00a"
      },
      "outputs": [],
      "source": [
        "def transform_func(inputs: dict) -> dict:\n",
        "    text = inputs[\"text\"]\n",
        "\n",
        "    # replace multiple new lines and multiple spaces with a single one\n",
        "    text = re.sub(r'(\\r\\n|\\r|\\n){2,}', r'\\n', text)\n",
        "    text = re.sub(r'[ \\t]+', ' ', text)\n",
        "\n",
        "    return {\"output_text\": text}"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "42dc1ac6",
      "metadata": {
        "hidden": true,
        "id": "42dc1ac6"
      },
      "source": [
        "Importantly, when we initialize the chain we do not send an llm as an argument. As you can imagine, not having an llm makes this chain's abilities much weaker than the example we saw earlier. However, as we will see next, combining this chain with other chains can give us highly desirable results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "286f7295",
      "metadata": {
        "hidden": true,
        "id": "286f7295"
      },
      "outputs": [],
      "source": [
        "clean_extra_spaces_chain = TransformChain(input_variables=[\"text\"], output_variables=[\"output_text\"], transform=transform_func)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "977bf11a",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "hidden": true,
        "id": "977bf11a",
        "outputId": "d1775672-cab6-4d52-8b31-cea3338e8b7a"
      },
      "outputs": [],
      "source": [
        "clean_extra_spaces_chain.run('A random text  with   some irregular spacing.\\n\\n\\n     Another one   here as well.')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b3f84cd0",
      "metadata": {
        "hidden": true,
        "id": "b3f84cd0"
      },
      "source": [
        "Great! Now things will get interesting.\n",
        "\n",
        "Say we want to use our chain to clean an input text and then paraphrase the input in a specific style, say a poet or a policeman. As we now know, the `TransformChain` does not use a llm so the styling will have to be done elsewhere. That's where our `LLMChain` comes in. We know about this chain already and we know that we can do cool things with smart prompting so let's take a chance!"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5b77042a",
      "metadata": {
        "hidden": true,
        "id": "5b77042a"
      },
      "source": [
        "First we will build the prompt template:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "73719a5d",
      "metadata": {
        "hidden": true,
        "id": "73719a5d"
      },
      "outputs": [],
      "source": [
        "template = \"\"\"Paraphrase this text:\n",
        "\n",
        "{output_text}\n",
        "\n",
        "In the style of a {style}.\n",
        "\n",
        "Paraphrase: \"\"\"\n",
        "prompt = PromptTemplate(input_variables=[\"style\", \"output_text\"], template=template)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "83b2ec83",
      "metadata": {
        "hidden": true,
        "id": "83b2ec83"
      },
      "source": [
        "And next, initialize our chain:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "48a067ab",
      "metadata": {
        "hidden": true,
        "id": "48a067ab"
      },
      "outputs": [],
      "source": [
        "style_paraphrase_chain = LLMChain(llm=llm, prompt=prompt, output_key='final_output')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2324005d",
      "metadata": {
        "hidden": true,
        "id": "2324005d"
      },
      "source": [
        "Great! Notice that the input text in the template is called 'output_text'. Can you guess why?\n",
        "\n",
        "We are going to pass the output of the `TransformChain` to the `LLMChain`!"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c5da4925",
      "metadata": {
        "hidden": true,
        "id": "c5da4925"
      },
      "source": [
        "Finally, we need to combine them both to work as one integrated chain. For that we will use `SequentialChain` which is our third generic chain building block."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "06f51f17",
      "metadata": {
        "hidden": true,
        "id": "06f51f17"
      },
      "outputs": [],
      "source": [
        "sequential_chain = SequentialChain(chains=[clean_extra_spaces_chain, style_paraphrase_chain], input_variables=['text', 'style'], output_variables=['final_output'])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7f0f51d8",
      "metadata": {
        "hidden": true,
        "id": "7f0f51d8"
      },
      "source": [
        "Our input is the langchain docs description of what chains are but dirty with some extra spaces all around."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "a8032489",
      "metadata": {
        "hidden": true,
        "id": "a8032489"
      },
      "outputs": [],
      "source": [
        "input_text = \"\"\"\n",
        "Chains allow us to combine multiple\n",
        "\n",
        "\n",
        "components together to create a single, coherent application.\n",
        "\n",
        "For example, we can create a chain that takes user input,       format it with a PromptTemplate,\n",
        "\n",
        "and then passes the formatted response to an LLM. We can build more complex chains by combining     multiple chains together, or by\n",
        "\n",
        "\n",
        "combining chains with other components.\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b2f55d21",
      "metadata": {
        "hidden": true,
        "id": "b2f55d21"
      },
      "source": [
        "We are all set. Time to get creative!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d507aa5c",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "hidden": true,
        "id": "d507aa5c",
        "outputId": "495641c4-0d7d-4723-f945-31a4d5307a46"
      },
      "outputs": [],
      "source": [
        "count_tokens(sequential_chain, {'text': input_text, 'style': 'a 90s rapper'})"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "60b52e19",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "60b52e19"
      },
      "source": [
        "### A note on langchain-hub"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "02f649da",
      "metadata": {
        "hidden": true,
        "id": "02f649da"
      },
      "source": [
        "`langchain-hub` is a sister library to `langchain`, where all the chains, agents and prompts are serialized for us to use."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "411500c2",
      "metadata": {
        "hidden": true,
        "id": "411500c2"
      },
      "outputs": [],
      "source": [
        "from langchain.chains import load_chain"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b375e5b7",
      "metadata": {
        "hidden": true,
        "id": "b375e5b7"
      },
      "source": [
        "Loading from langchain hub is as easy as finding the chain you want to load in the repository and then using `load_chain` with the corresponding path. We also have `load_prompt` and `initialize_agent`, but more on that later. Let's see how we can do this with our `LLMMathChain` we saw earlier:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fbe8748d",
      "metadata": {
        "hidden": true,
        "id": "fbe8748d"
      },
      "outputs": [],
      "source": [
        "llm_math_chain = load_chain('lc://chains/llm-math/chain.json')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ebcfe67c",
      "metadata": {
        "hidden": true,
        "id": "ebcfe67c"
      },
      "source": [
        "What if we want to change some of the configuration parameters? We can simply override it after loading:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d0d54233",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "d0d54233",
        "outputId": "e93fcb88-984e-4102-af9e-a5b6f3bf22f5"
      },
      "outputs": [],
      "source": [
        "llm_math_chain.verbose"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "074f8806",
      "metadata": {
        "hidden": true,
        "id": "074f8806"
      },
      "outputs": [],
      "source": [
        "llm_math_chain.verbose = False"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "465a6cbf",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "465a6cbf",
        "outputId": "fb34d899-b246-4ad5-ff82-80653b5f2103"
      },
      "outputs": [],
      "source": [
        "llm_math_chain.verbose"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2cc688ca",
      "metadata": {
        "hidden": true,
        "id": "2cc688ca"
      },
      "source": [
        "That's it for this example on chains.\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8d5ddfba",
      "metadata": {
        "heading_collapsed": true,
        "id": "8d5ddfba",
        "jp-MarkdownHeadingCollapsed": true
      },
      "source": [
        "# Conversational Memory with LangChain"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "hcqKO0aI6_PI",
      "metadata": {
        "hidden": true,
        "id": "hcqKO0aI6_PI"
      },
      "source": [
        "Conversational memory is how chatbots can respond to our queries in a chat-like manner. It enables a coherent conversation, and without it, every query would be treated as an entirely independent input without considering past interactions.\n",
        "\n",
        "The memory allows a _\"agent\"_ to remember previous interactions with the user. By default, agents are *stateless* — meaning each incoming query is processed independently of other interactions. The only thing that exists for a stateless agent is the current input, nothing else.\n",
        "\n",
        "There are many applications where remembering previous interactions is very important, such as chatbots. Conversational memory allows us to do that.\n",
        "\n",
        "In this notebook we'll explore this form of memory in the context of the LangChain library.\n",
        "\n",
        "We'll start by importing all of the libraries that we'll be using in this example."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "07c3fc35",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "07c3fc35",
        "outputId": "534f4158-b3bf-468e-85d8-80a4a5378168"
      },
      "outputs": [],
      "source": [
        "!pip install -qU tiktoken"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6e99e024",
      "metadata": {
        "hidden": true,
        "id": "6e99e024"
      },
      "outputs": [],
      "source": [
        "import inspect\n",
        "\n",
        "from getpass import getpass\n",
        "from langchain import OpenAI\n",
        "from langchain.chains import LLMChain, ConversationChain\n",
        "from langchain.chains.conversation.memory import (ConversationBufferMemory,\n",
        "                                                  ConversationSummaryMemory,\n",
        "                                                  ConversationBufferWindowMemory,\n",
        "                                                  ConversationKGMemory)\n",
        "from langchain.callbacks import get_openai_callback\n",
        "import tiktoken"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d09f87cb",
      "metadata": {
        "hidden": true,
        "id": "d09f87cb"
      },
      "source": [
        "To run this notebook, we will need to use an OpenAI LLM. Here we will setup the LLM we will use for the whole notebook, just input your openai api key when prompted."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e4498596",
      "metadata": {
        "hidden": true,
        "id": "e4498596"
      },
      "outputs": [],
      "source": [
        "llm = OpenAI(\n",
        "    temperature=0,\n",
        "    model_name='text-davinci-003'  # can be used with llms like 'gpt-3.5-turbo'\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dcdc08e6",
      "metadata": {
        "hidden": true,
        "id": "dcdc08e6"
      },
      "source": [
        "Later we will make use of a `count_tokens` utility function. This will allow us to count the number of tokens we are using for each call. We define it as so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4568c2ad",
      "metadata": {
        "hidden": true,
        "id": "4568c2ad"
      },
      "outputs": [],
      "source": [
        "def count_tokens(chain, query):\n",
        "    with get_openai_callback() as cb:\n",
        "        result = chain.run(query)\n",
        "        print(f'Spent a total of {cb.total_tokens} tokens')\n",
        "\n",
        "    return result"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "CnNF6i9r8RY_",
      "metadata": {
        "hidden": true,
        "id": "CnNF6i9r8RY_"
      },
      "source": [
        "Now let's dive into **Conversational Memory**."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "34f915b2",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "34f915b2"
      },
      "source": [
        "## What is memory?"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5b5492f8",
      "metadata": {
        "hidden": true,
        "id": "5b5492f8"
      },
      "source": [
        "**Definition**: Memory is an agent's capacity of remembering previous interactions with the user (think chatbots)\n",
        "\n",
        "The official definition of memory is the following:\n",
        "\n",
        "\n",
        "> By default, Chains and Agents are stateless, meaning that they treat each incoming query independently. In some applications (chatbots being a GREAT example) it is highly important to remember previous interactions, both at a short term but also at a long term level. The concept of “Memory” exists to do exactly that.\n",
        "\n",
        "\n",
        "As we will see, although this sounds really straightforward there are several different ways to implement this memory capability."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3343a0e2",
      "metadata": {
        "hidden": true,
        "id": "3343a0e2"
      },
      "source": [
        "Before we delve into the different memory modules that the library offers, we will introduce the chain we will be using for these examples: the `ConversationChain`."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6c9c13e9",
      "metadata": {
        "hidden": true,
        "id": "6c9c13e9"
      },
      "source": [
        "As always, when understanding a chain it is interesting to peek into its prompt first and then take a look at its `._call` method. As we saw in the chapter on chains, we can check out the prompt by accessing the `template` within the `prompt` attribute."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "96ff1ce3",
      "metadata": {
        "hidden": true,
        "id": "96ff1ce3"
      },
      "outputs": [],
      "source": [
        "conversation = ConversationChain(\n",
        "    llm=llm,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "90ad394d",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "90ad394d",
        "outputId": "53f9ae6c-8da8-4d94-9d9c-155215265768"
      },
      "outputs": [],
      "source": [
        "print(conversation.prompt.template)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9f8b1e0c",
      "metadata": {
        "hidden": true,
        "id": "9f8b1e0c"
      },
      "source": [
        "Interesting! So this chain's prompt is telling it to chat with the user and try to give truthful answers. If we look closely, there is a new component in the prompt that we didn't see when we were tinkering with the `LLMMathChain`: _history_. This is where our memory will come into play."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4a7e7770",
      "metadata": {
        "hidden": true,
        "id": "4a7e7770"
      },
      "source": [
        "What is this chain doing with this prompt? Let's take a look."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "43bfd2da",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "43bfd2da",
        "outputId": "37e6bb1f-f6e0-444f-f8f0-25bea70ce0e1"
      },
      "outputs": [],
      "source": [
        "print(inspect.getsource(conversation._call), inspect.getsource(conversation.apply))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "84e664af",
      "metadata": {
        "hidden": true,
        "id": "84e664af"
      },
      "source": [
        "Nothing really magical going on here, just a straightforward pass through an LLM. In fact, this chain inherits these methods directly from the `LLMChain` without any modification:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d8f4aa79",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "d8f4aa79",
        "outputId": "786f14ec-2485-4ada-c10a-260d446be064"
      },
      "outputs": [],
      "source": [
        "print(inspect.getsource(LLMChain._call), inspect.getsource(LLMChain.apply))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6aaa70bf",
      "metadata": {
        "hidden": true,
        "id": "6aaa70bf"
      },
      "source": [
        "So basically this chain combines an input from the user with the conversation history to generate a meaningful (and hopefully truthful) response."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "19f5172f",
      "metadata": {
        "hidden": true,
        "id": "19f5172f"
      },
      "source": [
        "Now that we've understood the basics of the chain we'll be using, we can get into memory. Let's dive in!"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0f1a33f6",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "0f1a33f6"
      },
      "source": [
        "## Memory types"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4d732b7a",
      "metadata": {
        "hidden": true,
        "id": "4d732b7a"
      },
      "source": [
        "In this section we will review several memory types and analyze the pros and cons of each one, so you can choose the best one for your use case."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "04d70642",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "04d70642"
      },
      "source": [
        "### Memory type #1: ConversationBufferMemory"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "53d3cb2b",
      "metadata": {
        "hidden": true,
        "id": "53d3cb2b"
      },
      "source": [
        "The `ConversationBufferMemory` does just what its name suggests: it keeps a buffer of the previous conversation excerpts as part of the context in the prompt."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d80a974a",
      "metadata": {
        "hidden": true,
        "id": "d80a974a"
      },
      "source": [
        "**Key feature:** _the conversation buffer memory keeps the previous pieces of conversation completely unmodified, in their raw form._"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2267f1f0",
      "metadata": {
        "hidden": true,
        "id": "2267f1f0"
      },
      "outputs": [],
      "source": [
        "conversation_buf = ConversationChain(\n",
        "    llm=llm,\n",
        "    memory=ConversationBufferMemory()\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "lseziAMcAyvX",
      "metadata": {
        "hidden": true,
        "id": "lseziAMcAyvX"
      },
      "source": [
        "We pass a user prompt the the `ConversationBufferMemory` like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "M0cwooC5A5Id",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "M0cwooC5A5Id",
        "outputId": "7435b891-c5d6-4f5b-cf43-1e52f5c4a0a8"
      },
      "outputs": [],
      "source": [
        "conversation_buf(\"Good morning AI!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xlKINTFYA9eo",
      "metadata": {
        "hidden": true,
        "id": "xlKINTFYA9eo"
      },
      "source": [
        "This one call used a total of `85` tokens, but we can't see that from the above. If we'd like to count the number of tokens being used we just pass our conversation chain object and the message we'd like to input via the `count_tokens` function we defined earlier:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d1bd5a88",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 106
        },
        "hidden": true,
        "id": "d1bd5a88",
        "outputId": "9f5ef0a9-0f66-4f1f-b1cd-784dff875e97"
      },
      "outputs": [],
      "source": [
        "count_tokens(\n",
        "    conversation_buf,\n",
        "    \"My interest here is to explore the potential of integrating Large Language Models with external knowledge\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "146170ca",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 106
        },
        "hidden": true,
        "id": "146170ca",
        "outputId": "6e1c61c7-dce6-466f-89bf-5093136e3456"
      },
      "outputs": [],
      "source": [
        "count_tokens(\n",
        "    conversation_buf,\n",
        "    \"I just want to analyze the different possibilities. What can you think of?\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3e15411a",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "hidden": true,
        "id": "3e15411a",
        "outputId": "5c2611ed-c882-4123-91af-55ee27a7b5d1"
      },
      "outputs": [],
      "source": [
        "count_tokens(\n",
        "    conversation_buf,\n",
        "    \"Which data source types could be used to give context to the model?\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3352cc48",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "hidden": true,
        "id": "3352cc48",
        "outputId": "f59b94e6-3a86-4015-d75a-f484f5c51fc9"
      },
      "outputs": [],
      "source": [
        "count_tokens(\n",
        "    conversation_buf,\n",
        "    \"What is my aim again?\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "431b74ff",
      "metadata": {
        "hidden": true,
        "id": "431b74ff"
      },
      "source": [
        "Our LLM with `ConversationBufferMemory` can clearly remember earlier interactions in the conversation. Let's take a closer look to how the LLM is saving our previous conversation. We can do this by accessing the `.buffer` attribute for the `.memory` in our chain."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "984afd09",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "984afd09",
        "outputId": "1aebaed0-59be-4c0d-9796-581da73fefc3"
      },
      "outputs": [],
      "source": [
        "print(conversation_buf.memory.buffer)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4570267d",
      "metadata": {
        "hidden": true,
        "id": "4570267d"
      },
      "source": [
        "Nice! So every piece of our conversation has been explicitly recorded and sent to the LLM in the prompt."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "acf1a90b",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "acf1a90b"
      },
      "source": [
        "### Memory type #2: ConversationSummaryMemory"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "01f61fe9",
      "metadata": {
        "hidden": true,
        "id": "01f61fe9"
      },
      "source": [
        "The problem with the `ConversationBufferMemory` is that as the conversation progresses, the token count of our context history adds up. This is problematic because we might max out our LLM with a prompt that is too large to be processed."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0516c7d4",
      "metadata": {
        "hidden": true,
        "id": "0516c7d4"
      },
      "source": [
        "Enter `ConversationSummaryMemory`.\n",
        "\n",
        "Again, we can infer from the name what is going on.. we will keep a summary of our previous conversation snippets as our history. How will we summarize these? LLM to the rescue."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "86b0a905",
      "metadata": {
        "hidden": true,
        "id": "86b0a905"
      },
      "source": [
        "**Key feature:** _the conversation summary memory keeps the previous pieces of conversation in a summarized form, where the summarization is performed by an LLM._"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0ea6050c",
      "metadata": {
        "hidden": true,
        "id": "0ea6050c"
      },
      "source": [
        "In this case we need to send the llm to our memory constructor to power its summarization ability."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f33a16a7",
      "metadata": {
        "hidden": true,
        "id": "f33a16a7"
      },
      "outputs": [],
      "source": [
        "conversation_sum = ConversationChain(\n",
        "    llm=llm,\n",
        "    memory=ConversationSummaryMemory(llm=llm)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b64c4896",
      "metadata": {
        "hidden": true,
        "id": "b64c4896"
      },
      "source": [
        "When we have an llm, we always have a prompt ;) Let's see what's going on inside our conversation summary memory:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c476824d",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "c476824d",
        "outputId": "6fe5e7d5-573f-43ff-933f-6dd48ed7b94e"
      },
      "outputs": [],
      "source": [
        "print(conversation_sum.memory.prompt.template)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "df90cdf3",
      "metadata": {
        "hidden": true,
        "id": "df90cdf3"
      },
      "source": [
        "Cool! So each new interaction is summarized and appended to a running summary as the memory of our chain. Let's see how this works in practice!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "34343665",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "hidden": true,
        "id": "34343665",
        "outputId": "e2c577f4-bc0d-4aa6-f3a1-fffcabbc01dc"
      },
      "outputs": [],
      "source": [
        "# without count_tokens we'd call `conversation_sum(\"Good morning AI!\")`\n",
        "# but let's keep track of our tokens:\n",
        "count_tokens(\n",
        "    conversation_sum,\n",
        "    \"Good morning AI!\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b757bba3",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "hidden": true,
        "id": "b757bba3",
        "outputId": "9de1823a-0dfe-45ff-fadc-26eff6fdce99"
      },
      "outputs": [],
      "source": [
        "count_tokens(\n",
        "    conversation_sum,\n",
        "    \"My interest here is to explore the potential of integrating Large Language Models with external knowledge\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d0a373e2",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "hidden": true,
        "id": "d0a373e2",
        "outputId": "9ffca0fd-60a6-4cb7-df69-33182aaf1317"
      },
      "outputs": [],
      "source": [
        "count_tokens(\n",
        "    conversation_sum,\n",
        "    \"I just want to analyze the different possibilities. What can you think of?\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2e286f0d",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "hidden": true,
        "id": "2e286f0d",
        "outputId": "9558ef92-5f9c-4818-be8b-1e7e6ec19864"
      },
      "outputs": [],
      "source": [
        "count_tokens(\n",
        "    conversation_sum,\n",
        "    \"Which data source types could be used to give context to the model?\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "891180f2",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "hidden": true,
        "id": "891180f2",
        "outputId": "2f2a114a-2a29-40f1-eb35-5e4db8dbe968"
      },
      "outputs": [],
      "source": [
        "count_tokens(\n",
        "    conversation_sum,\n",
        "    \"What is my aim again?\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2d768e44",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "2d768e44",
        "outputId": "76b7fc29-12e1-43b5-f259-b93b204bce68"
      },
      "outputs": [],
      "source": [
        "print(conversation_sum.memory.buffer)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0dd35c8c",
      "metadata": {
        "hidden": true,
        "id": "0dd35c8c"
      },
      "source": [
        "You might be wondering.. if the aggregate token count is greater in each call here than in the buffer example, why should we use this type of memory? Well, if we check out buffer we will realize that although we are using more tokens in each instance of our conversation, our final history is shorter. This will enable us to have many more interactions before we reach our prompt's max length, making our chatbot more robust to longer conversations.\n",
        "\n",
        "We can count the number of tokens being used (without making a call to OpenAI) using the `tiktoken` tokenizer like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "nzijj4RZFX3I",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "nzijj4RZFX3I",
        "outputId": "6a1df09c-01ab-4607-ff49-2096d22099f4"
      },
      "outputs": [],
      "source": [
        "# initialize tokenizer\n",
        "tokenizer = tiktoken.encoding_for_model('text-davinci-003')\n",
        "\n",
        "# show number of tokens for the memory used by each memory type\n",
        "print(\n",
        "    f'Buffer memory conversation length: {len(tokenizer.encode(conversation_buf.memory.buffer))}\\n'\n",
        "    f'Summary memory conversation length: {len(tokenizer.encode(conversation_sum.memory.buffer))}'\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2bab0c09",
      "metadata": {
        "hidden": true,
        "id": "2bab0c09"
      },
      "source": [
        "_Practical Note: the `text-davinci-003` and `gpt-3.5-turbo` models [have](https://platform.openai.com/docs/api-reference/completions/create#completions/create-max_tokens) a large max tokens count of 4096 tokens between prompt and answer._"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "494830ea",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "494830ea"
      },
      "source": [
        "### Memory type #3: ConversationBufferWindowMemory"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "00762844",
      "metadata": {
        "hidden": true,
        "id": "00762844"
      },
      "source": [
        "Another great option for these cases is the `ConversationBufferWindowMemory` where we will be keeping a few of the last interactions in our memory but we will intentionally drop the oldest ones - short-term memory if you'd like. Here the aggregate token count **and** the per-call token count will drop noticeably. We will control this window with the `k` parameter."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "206a5915",
      "metadata": {
        "hidden": true,
        "id": "206a5915"
      },
      "source": [
        "**Key feature:** _the conversation buffer window memory keeps the latest pieces of the conversation in raw form_"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "45be373a",
      "metadata": {
        "hidden": true,
        "id": "45be373a"
      },
      "outputs": [],
      "source": [
        "conversation_bufw = ConversationChain(\n",
        "    llm=llm,\n",
        "    memory=ConversationBufferWindowMemory(k=1)\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fc4dd8a0",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "hidden": true,
        "id": "fc4dd8a0",
        "outputId": "c4ec1cc8-f218-4f7b-e27e-f5fb73e59228"
      },
      "outputs": [],
      "source": [
        "count_tokens(\n",
        "    conversation_bufw,\n",
        "    \"Good morning AI!\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b9992e8d",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "hidden": true,
        "id": "b9992e8d",
        "outputId": "ac7ae1af-2329-4766-ac5e-8fce24a1d272"
      },
      "outputs": [],
      "source": [
        "count_tokens(\n",
        "    conversation_bufw,\n",
        "    \"My interest here is to explore the potential of integrating Large Language Models with external knowledge\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3f2e98d9",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "hidden": true,
        "id": "3f2e98d9",
        "outputId": "dc60726a-4be2-480f-892b-443da9b2859e"
      },
      "outputs": [],
      "source": [
        "count_tokens(\n",
        "    conversation_bufw,\n",
        "    \"I just want to analyze the different possibilities. What can you think of?\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "a2a8d062",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 106
        },
        "hidden": true,
        "id": "a2a8d062",
        "outputId": "dbb27cf0-2e87-41d0-a733-68921d250481"
      },
      "outputs": [],
      "source": [
        "count_tokens(\n",
        "    conversation_bufw,\n",
        "    \"Which data source types could be used to give context to the model?\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ff199a3f",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "hidden": true,
        "id": "ff199a3f",
        "outputId": "81573cf0-7f39-4a8c-8ccd-e79cd80f2523"
      },
      "outputs": [],
      "source": [
        "count_tokens(\n",
        "    conversation_bufw,\n",
        "    \"What is my aim again?\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f5f59f77",
      "metadata": {
        "hidden": true,
        "id": "f5f59f77"
      },
      "source": [
        "As we can see, it effectively 'fogot' what we talked about in the first interaction. Let's see what it 'remembers'. Given that we set k to be `1`, we would expect it remembers only the last interaction."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6b354c8d",
      "metadata": {
        "hidden": true,
        "id": "6b354c8d"
      },
      "source": [
        "We need to access a special method here since, in this memory type, the buffer is first passed through this method to be sent later to the llm."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "85266406",
      "metadata": {
        "hidden": true,
        "id": "85266406"
      },
      "outputs": [],
      "source": [
        "bufw_history = conversation_bufw.memory.load_memory_variables(\n",
        "    inputs=[]\n",
        ")['history']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5904ae2a",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "5904ae2a",
        "outputId": "bd0aa797-7a43-4af5-a531-209aa6272dd4"
      },
      "outputs": [],
      "source": [
        "print(bufw_history)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ae8b937d",
      "metadata": {
        "hidden": true,
        "id": "ae8b937d"
      },
      "source": [
        "Makes sense.\n",
        "\n",
        "On the plus side, we are shortening our conversation length when compared to buffer memory _without_ a window:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9fbb50fe",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "9fbb50fe",
        "outputId": "c35dca36-a7c7-4d61-da19-c28173fa8319"
      },
      "outputs": [],
      "source": [
        "print(\n",
        "    f'Buffer memory conversation length: {len(tokenizer.encode(conversation_buf.memory.buffer))}\\n'\n",
        "    f'Summary memory conversation length: {len(tokenizer.encode(conversation_sum.memory.buffer))}\\n'\n",
        "    f'Buffer window memory conversation length: {len(tokenizer.encode(bufw_history))}'\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "69842cc1",
      "metadata": {
        "hidden": true,
        "id": "69842cc1"
      },
      "source": [
        "_Practical Note: We are using `k=2` here for illustrative purposes, in most real world applications you would need a higher value for k._"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2aea5fc8",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "2aea5fc8"
      },
      "source": [
        "### More memory types!"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "daeb5162",
      "metadata": {
        "hidden": true,
        "id": "daeb5162"
      },
      "source": [
        "Given that we understand memory already, we will present a few more memory types here and hopefully a brief description will be enough to understand their underlying functionality."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f0365333",
      "metadata": {
        "hidden": true,
        "id": "f0365333"
      },
      "source": [
        "#### ConversationSummaryBufferMemory"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "317f298e",
      "metadata": {
        "hidden": true,
        "id": "317f298e"
      },
      "source": [
        "**Key feature:** _the conversation summary memory keeps a summary of the earliest pieces of conversation while retaining a raw recollection of the latest interactions._"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "57ef5c8b",
      "metadata": {
        "hidden": true,
        "id": "57ef5c8b"
      },
      "source": [
        "#### ConversationKnowledgeGraphMemory"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "40248f03",
      "metadata": {
        "hidden": true,
        "id": "40248f03"
      },
      "source": [
        "This is a super cool memory type that was introduced just [recently](https://twitter.com/LangChainAI/status/1625158388824043522). It is based on the concept of a _knowledge graph_ which recognizes different entities and connects them in pairs with a predicate resulting in (subject, predicate, object) triplets. This enables us to compress a lot of information into highly significant snippets that can be fed into the model as context. If you want to understand this memory type in more depth you can check out [this](https://apex974.com/articles/explore-langchain-support-for-knowledge-graph) blogpost."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "91952cd1",
      "metadata": {
        "hidden": true,
        "id": "91952cd1"
      },
      "source": [
        "**Key feature:** _the conversation knowledge graph memory keeps a knowledge graph of all the entities that have been mentioned in the interactions together with their semantic relationships._"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "02241bc3",
      "metadata": {
        "hidden": true,
        "id": "02241bc3"
      },
      "outputs": [],
      "source": [
        "# you may need to install this library\n",
        "# !pip install -qU networkx"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c5f10a89",
      "metadata": {
        "hidden": true,
        "id": "c5f10a89"
      },
      "outputs": [],
      "source": [
        "conversation_kg = ConversationChain(\n",
        "    llm=llm,\n",
        "    memory=ConversationKGMemory(llm=llm)\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "65957fe2",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "hidden": true,
        "id": "65957fe2",
        "outputId": "c9561a4a-412a-4d92-865d-9e81a09bb101"
      },
      "outputs": [],
      "source": [
        "count_tokens(\n",
        "    conversation_kg,\n",
        "    \"My name is human and I like mangoes!\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "74054534",
      "metadata": {
        "hidden": true,
        "id": "74054534"
      },
      "source": [
        "The memory keeps a knowledge graph of everything it learned so far."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5a8c54fb",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "hidden": true,
        "id": "5a8c54fb",
        "outputId": "adf96679-087b-4b77-c00d-9bf9e98f9278"
      },
      "outputs": [],
      "source": [
        "conversation_kg.memory.kg.get_triples()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e1a1ca15",
      "metadata": {
        "hidden": true,
        "id": "e1a1ca15"
      },
      "source": [
        "#### ConversationEntityMemory"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "41e9aeaf",
      "metadata": {
        "hidden": true,
        "id": "41e9aeaf"
      },
      "source": [
        "**Key feature:** _the conversation entity memory keeps a recollection of the main entities that have been mentioned, together with their specific attributes._"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2900a385",
      "metadata": {
        "hidden": true,
        "id": "2900a385"
      },
      "source": [
        "The way this works is quite similar to the `ConversationKnowledgeGraphMemory`, you can refer to the [docs](https://langchain.readthedocs.io/en/latest/modules/memory/examples/entity_summary_memory.html) if you want to see it in action."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d45112bd",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "d45112bd"
      },
      "source": [
        "## What else can we do with memory?"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "78296bff",
      "metadata": {
        "hidden": true,
        "id": "78296bff"
      },
      "source": [
        "There are several cool things we can do with memory in langchain. We can:\n",
        "* implement our own custom memory module\n",
        "* use multiple memory modules in the same chain\n",
        "* combine agents with memory and other tools\n",
        "\n",
        "If this piques your interest, we suggest you to go take a look at the memory [how-to](https://langchain.readthedocs.io/en/latest/modules/memory/how_to_guides.html) section in the docs!"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7abe2121",
      "metadata": {
        "heading_collapsed": true,
        "hidden": true,
        "id": "7abe2121"
      },
      "source": [
        "## Extra Material: Token Counter"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6403e8fb",
      "metadata": {
        "hidden": true,
        "id": "6403e8fb"
      },
      "source": [
        "This is an additional piece of material alongside the [LangChain Handbook notebook on Conversational Memory](https://github.com/pinecone-io/examples/blob/master/generation/langchain/handbook/03-langchain-conversational-memory.ipynb).\n",
        "\n",
        "In this notebook we will count the number of tokens used in a conversation for different conversational memory types.\n",
        "\n",
        "We begin by installing the required libraries:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1d50a3cd",
      "metadata": {
        "hidden": true,
        "id": "1d50a3cd"
      },
      "outputs": [],
      "source": [
        "# !pip install -qU langchain openai transformers"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "853ad81e",
      "metadata": {
        "hidden": true,
        "id": "853ad81e"
      },
      "source": [
        "Import required libraries and objects:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ac0540dc",
      "metadata": {
        "hidden": true,
        "id": "ac0540dc"
      },
      "outputs": [],
      "source": [
        "from getpass import getpass\n",
        "\n",
        "import openai\n",
        "from langchain import OpenAI\n",
        "from langchain.chains import LLMChain, ConversationChain\n",
        "from langchain.chains.conversation.memory import (\n",
        "    ConversationBufferMemory,\n",
        "    ConversationSummaryMemory,\n",
        "    ConversationBufferWindowMemory,\n",
        "    ConversationSummaryBufferMemory\n",
        ")\n",
        "from langchain.callbacks import get_openai_callback\n",
        "from tqdm.auto import tqdm"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c2f3cd4b",
      "metadata": {
        "hidden": true,
        "id": "c2f3cd4b"
      },
      "source": [
        "To run the notebook we'll use OpenAI's `gpt-3.5-turbo` model. We initialize it via LangChain like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "840dab44",
      "metadata": {
        "hidden": true,
        "id": "840dab44",
        "outputId": "3cca0a31-4250-45d0-a987-b764f0a3d919"
      },
      "outputs": [],
      "source": [
        "llm = OpenAI(\n",
        "    temperature=0,\n",
        "    model_name='gpt-3.5-turbo'  # can be used with llms like 'text-davinci-003'\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6d3ae122",
      "metadata": {
        "hidden": true,
        "id": "6d3ae122"
      },
      "source": [
        "To count the number of tokens used during each call we will define a `count_tokens` function:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e1510396",
      "metadata": {
        "hidden": true,
        "id": "e1510396"
      },
      "outputs": [],
      "source": [
        "def count_tokens(chain, query):\n",
        "    with get_openai_callback() as cb:\n",
        "        result = chain.run(query)\n",
        "    return {\n",
        "        'result': result,\n",
        "        'token_count': cb.total_tokens\n",
        "    }"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7eaeca62",
      "metadata": {
        "hidden": true,
        "id": "7eaeca62"
      },
      "source": [
        "Let's define the conversation function:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6f76a39b",
      "metadata": {
        "hidden": true,
        "id": "6f76a39b"
      },
      "outputs": [],
      "source": [
        "queries = [\n",
        "    \"Good morning AI?\",\n",
        "    \"\"\"My interest here is to explore the potential of integrating Large\n",
        "    Language Models with external knowledge\"\"\",\n",
        "    \"I just want to analyze the different possibilities. What can you think of?\",\n",
        "    \"What about the use of retrieval augmentation, can that be used as well?\",\n",
        "    \"\"\"That's very interesting, can you tell me more about this? Like what\n",
        "    systems would I use to store the information and retrieve relevant info?\"\"\",\n",
        "    \"\"\"Okay that's cool, I've been hearing about 'vector databases', are they\n",
        "    relevant in this context?\"\"\",\n",
        "    \"\"\"Okay that's useful, but how do I go from my external knowledge to\n",
        "    creating these 'vectors'? I have no idea how text can become a vector?\"\"\",\n",
        "    \"\"\"Well I don't think I'd be using word embeddings right? If I wanted to\n",
        "    store my documents in this vector database, I suppose I would need to\n",
        "    transform the documents into vectors? Maybe I can use the 'sentence\n",
        "    embeddings' for this, what do you think?\"\"\",\n",
        "    \"\"\"Can sentence embeddings only represent sentences of text? That seems\n",
        "    kind of small to capture any meaning from a document? Is there any approach\n",
        "    that can encode at least a paragraph of text?\"\"\",\n",
        "    \"\"\"Huh, interesting. I do remember reading something about 'mpnet' or\n",
        "    'minilm' sentence 'transformer' models that could encode small to\n",
        "    medium sized paragraphs. Am I wrong about this?\"\"\",\n",
        "    \"\"\"Ah that's great to hear, do you happen to know how much text I can feed\n",
        "    into these types of models?\"\"\",\n",
        "    \"\"\"I've never heard of hierarchical embeddings, could you explain those in\n",
        "    more detail?\"\"\",\n",
        "    \"\"\"So is it like you have a transformer model or something else that creates\n",
        "    sentence level embeddings, then you feed all of the sentence level\n",
        "    embeddings into another separate neural network that knows how to merge\n",
        "    multiple sentence embeddings into a single embedding?\"\"\",\n",
        "    \"\"\"Could you explain this process step by step from start to finish? Explain\n",
        "    like I'm very new to this space, assume I don't have much prior knowledge\n",
        "    of embeddings, neural nets, etc\"\"\",\n",
        "    \"\"\"Awesome thanks! Are there any popular 'heirarchical neural network'\n",
        "    models that I can look up? Or maybe just the second stage that creates the\n",
        "    hierarchical embeddings?\"\"\",\n",
        "    \"It seems like these HAN models are quite old, is there anything more recent?\",\n",
        "    \"Can you explain the difference between transformer-XL and longformer?\",\n",
        "    \"How much text can be encoded by each of these models?\",\n",
        "    \"\"\"Okay very interesting, so before returning to earlier in the conversation.\n",
        "    I understand now that there are a lot of different transformer (and not\n",
        "    transformer) based models for creating the embeddings from vectors. Is that\n",
        "    correct?\"\"\",\n",
        "    \"\"\"Perfect, so I understand text can be encoded into these embeddings. But\n",
        "    what then? Once I have my embeddings what do I do?\"\"\",\n",
        "    \"\"\"I'd like to use these embeddings to help a chatbot or a question-answering\n",
        "    system answer questions with help from this external knowledge base. I\n",
        "    suppose this would come under information retrieval? Could you explain that\n",
        "    process in a little more detail?\"\"\",\n",
        "    \"\"\"Okay great, that sounds like what I'm hoping to do. When you say the\n",
        "    'chatbot or question-answering system generates an embedding', what do you\n",
        "    mean exactly?\"\"\",\n",
        "    \"\"\"Ah okay, I understand, so it isn't the 'chatbot' model specifically\n",
        "    creating the embedding right? That's how I understood your earlier comment.\n",
        "    It seems more like there is a separate embedding model? And that encodes\n",
        "    the query, then we retrieve the set of relevant documents from the\n",
        "    external knowledge base? How is that information then used by the chatbot\n",
        "    or question-answering system exactly?\"\"\",\n",
        "    \"\"\"Okay but how is the information provided to the chatbot or\n",
        "    question-answering system?\"\"\",\n",
        "    \"\"\"So the retrieved information is given to the chatbot / QA system as plain\n",
        "    text? But then how do we pass in the original query? How can the system\n",
        "    distinguish between a user's query and all of this additional information?\"\"\",\n",
        "    \"\"\"That doesn't seem correct to me, my question is — if we are giving the\n",
        "    chatbot / QA system the user's query AND retrieved information from an\n",
        "    external knowledge base, and it's all fed into the model as plain text,\n",
        "    how does the model know what part of the plain text is a query vs. retrieved\n",
        "    information?\"\"\",\n",
        "    \"\"\"Yes I get that, but in the text passed to the model, how do we identify\n",
        "    user prompt vs retrieved information?\"\"\"\n",
        "\n",
        "]\n",
        "\n",
        "def talk(conversation_chain):\n",
        "    tokens_used = []\n",
        "    # we loop through the conversation above, counting token usage as we go\n",
        "    for user_query in tqdm(queries):\n",
        "        try:\n",
        "            res = count_tokens(conversation_chain, user_query)\n",
        "            tokens_used.append(res['token_count'])\n",
        "        except openai.error.InvalidRequestError:\n",
        "            # we hit the token limit of the model, so break\n",
        "            break\n",
        "    return tokens_used"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "690acab7",
      "metadata": {
        "hidden": true,
        "id": "690acab7"
      },
      "source": [
        "Create set of conversation chains that we'll be using:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3cec13a1",
      "metadata": {
        "hidden": true,
        "id": "3cec13a1"
      },
      "outputs": [],
      "source": [
        "conversation_chains = {\n",
        "    'ConversationBufferMemory': ConversationChain(\n",
        "        llm=llm, memory=ConversationBufferMemory()\n",
        "    ),\n",
        "    'ConversationSummaryMemory': ConversationChain(\n",
        "        llm=llm, memory=ConversationSummaryMemory(llm=llm)\n",
        "    ),\n",
        "    'ConversationBufferWindowMemory(k=6)': ConversationChain(\n",
        "        llm=llm, memory=ConversationBufferWindowMemory(k=6)\n",
        "    ),\n",
        "    'ConversationBufferWindowMemory(k=12)': ConversationChain(\n",
        "        llm=llm, memory=ConversationBufferWindowMemory(k=12)\n",
        "    ),\n",
        "    'ConversationSummaryBufferMemory(k=6)': ConversationChain(\n",
        "        llm=llm, memory=ConversationSummaryBufferMemory(\n",
        "            llm=llm,\n",
        "            max_token_limit=650\n",
        "        )\n",
        "    ),\n",
        "    'ConversationSummaryBufferMemory(k=12)': ConversationChain(\n",
        "        llm=llm, memory=ConversationSummaryBufferMemory(\n",
        "            llm=llm,\n",
        "            max_token_limit=1_300\n",
        "        )\n",
        "    )\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "19cce890",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 441,
          "referenced_widgets": [
            "2be4d4a7e596437db88f541b8d34bc53",
            "36525328bdf1428fae13136f8a0bc7bb",
            "bf6a1f2dc2c64d5991e9068e84b650b0",
            "0dbec763335c48fcb2e2961fc29fc1b8",
            "b88a4fd2be2f493190aeb316912df69b",
            "4d9d241c61c14e37832d48e97f5b645a",
            "7d017ac6d0874d95a3947746f68d6108",
            "7743eca3ee7342ad86537e0c16e29b05",
            "5c5d01ecf397463c939e78951b96a7c5",
            "70d01f9e39ab4d84a7ca80971033ef5c",
            "367c66d5b6014e0db51334e6ed05becc",
            "b177914703a847479f2e5b876dbdf4f7",
            "ded0b567e5f944cf9e817ae7b86fe93c",
            "8708bdb3db13488bb097736bd4d5f316",
            "026aaf0f075f4d44b1fa22a91274e975",
            "47a83c8c9f4c4ba0adf125421f213d70",
            "7645bb2b57804bcca7d667267ac33bd0",
            "885c15fafa85427dac68dc0a38c7a896",
            "aaabd7206abd43ed885fb6cde41375c5",
            "2d3db1292fa14d6780d79eea806f3541",
            "baa29f094aec4c16868f6f9affc241cd",
            "39e5c5d2fc574d31919310ea8c1c9fc8",
            "2cd828e3928f45cc8cf6e0050cd4f6a7",
            "04daff3502b947c7887ac4d0f56fd70a",
            "b9d86a3e1fe549bc93129f9235552482",
            "02add01c37654a538f13010d7c71e753",
            "c154bffb05d740709b59a5921b4a2c3d",
            "ab269d223f0a4f128b2bf565dc4aa02c",
            "42467f81fade4b58b59b0576b5312d64",
            "acdb823e0b0c4a169526205fb6416ac8",
            "8329f7d3193441f9bd9e1b2f2cda1dcc",
            "6f4a826a519649c7842697dc52a02b3a",
            "6493e6f0939f41eb8dea255ca944c57a",
            "86387f7978d24544bc10cae2c488928a",
            "d9c9b754057946f8b18f686366fac93f",
            "b082e24b6df0495f949aa8d03d0dc519",
            "0631309faf8847159f4dc15618fedab6",
            "187cb0cc5b0849b3af5a54a5ad8393cd",
            "d684d579ad0b45c189bbb9de17a59a81",
            "c25f23a018044eb0b074ba8faec9ebb5",
            "cb98068b72274befb65d34ce6ded8b15",
            "356eccd305c44c92b3a768a7a497cd5e",
            "1ac2a1b16ad241b993bcf97955e81577",
            "35c22ad0b9074e408d58fcac08605e85",
            "4455e527a2744efc88c42f1cbec1536e",
            "0d047388c6b5438188a16a6cfeefbf23",
            "be219119da0d484f92931324f6746f95",
            "031b1285627e4865bad1199ee26906f1",
            "3cc548e94bfd4c96b8ede3c713a6ca2e",
            "129cbb406d9645cd814c34aac2b647ee",
            "c9a8a0c6aea54aa79c2de4ff3ccf77e8",
            "67cda98e1f2a48d5a8f3b54ff4f8f64f",
            "fa56f77bd85b4e459ae8cb86759164eb",
            "8865b266def842c181b6245f0e91b26c",
            "cd623dfbc84045eea812063c7a0c1dcc",
            "0cb7de9161b2409aa38b1860eb8cd990",
            "e30e6c6d98c94f4aa86c6161e12b3c98",
            "7ebf97578b32483b91b69d0d05ffa5a2",
            "202d247919764f65b361bcef75f5642a",
            "c361fca427d44e8cb0271fe2b6f4302e",
            "5e808dbcc14946538b1584192e483e51",
            "2707427a681a441cb4defca3d34bb748",
            "9f2d7cead4cc434ea935a5ec1cf1e47b",
            "2522cf65c1ba426ba27f030acc0633d9",
            "a772967ef93844d0b00f027ca4134c19",
            "a02cc655fd6a4e3095f2aa711a2c9a13",
            "79826df9c5cc404f95d4ce8ebcfa20ff",
            "30ff2b3d2b2448f3b0f68d4714684db3",
            "85b05228efb54ab5a5395096bc1e699e",
            "4b6924fee3fd412c8ee406fba3e6d4b4",
            "f6e785ee417141cb803dfd7f03254f79",
            "9a4d6fa52f6f474b9f8de85e4a8b3edf",
            "0436cae6211f4cd8a2ed5990f02cbdb8",
            "495465f82dc242f49b4c146b6458e5e4",
            "ed9cc4b3cfc04062b38047a367ba528b",
            "d08875d9c0744100ad7c3ced1aeeb2e3",
            "d3ec6ba8d29f41bbaf62f07a138226a7",
            "9dc25233e04e4b8cb3a6fa79f7eb1e23",
            "816e625042194436af8a193436f51e26",
            "b367539fab28453698f549eb286c98f8",
            "84b54b8074a04e6e85540d4faf5e27d5",
            "20dcc9c584dd47789c0b9297b4973b18",
            "6d5bc0a26699408cb9dbeaf831c9949d",
            "ca96a4cad9d945ccba9f1a9bf30a593d",
            "8b3875c1e821407c8118dba150838b31",
            "a65ecfbfb7e34465978fe2799e70319a",
            "6bd35b95ef114e4b8b57fc49757381a8",
            "87d981713a024b18b65ecb1c04b26bf2",
            "2ff386b5a15b43f4b75ba4c6fa6959a7",
            "76df0865ad634f7a8061e96d8f5b80c7",
            "ed889a0e039646afb9ac267b32507df3",
            "f7984859eacc41c89ef33526efa1df0b",
            "9e19e310ce044564842537965ee66841",
            "6c13c9ebeb474c66907262c9573311d0",
            "aeeb99e1086840c8b7d0807ccd0b057b",
            "c0cf94856df84c12987b54b5608c258d",
            "6de6c6d6f201416d850455f96b94407f",
            "6c883833b99448fba7ff6c8b2cc30aee",
            "3169a365eb504f838c6726a93adf1a93",
            "6193595b5824404b8e6c9428612d67ee",
            "c423831fe1e543458ab1fc16d68dbc42",
            "79b582179bea4d299eaeabb6b54ad49b",
            "073d4b875d57463398fc936f9272ffa1",
            "a79b233bb44a4e699f7241a0bee9fe42",
            "8e45498c5bc748479c8dcd1e615e8add",
            "aa288fc687404376ab55bdf5dd036ce1",
            "fd2a27dff42f43b3b9c25d678d6c32f0",
            "470487e6768148049b091513538236bb",
            "e9443b0de3d34fe5b9f51b2a419acff0",
            "c49af1d9b28741299e25dd2646099e23",
            "aa2e6c66886e485789c3174e1edb32a9",
            "c553d848c49947d88d668b7421f26620"
          ]
        },
        "hidden": true,
        "id": "19cce890",
        "outputId": "1a29133b-e7f2-4ac3-a80a-2dd674e3d8c7",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "counts = {}\n",
        "# loop through each of our memory types above\n",
        "for key, chain in conversation_chains.items():\n",
        "    print(key)\n",
        "    counts[key] = talk(chain)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7fa32487",
      "metadata": {
        "hidden": true,
        "id": "7fa32487"
      },
      "outputs": [],
      "source": [
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.figure(figsize=(12,8))\n",
        "max_tokens = 4096\n",
        "\n",
        "colors = [\"#1c17ff\", \"#738FAB\", \"#f77f00\", \"#fcbf49\", \"#38c172\", \"#4dc0b5\"]\n",
        "\n",
        "for i, (key, count) in enumerate(counts.items()):\n",
        "    color = colors[i]\n",
        "    sns.lineplot(\n",
        "        x=range(1, len(count)+1),\n",
        "        y=count,\n",
        "        label=key,\n",
        "        color=color\n",
        "    )\n",
        "    if max_tokens in count:\n",
        "        plt.plot(\n",
        "            len(count), max_tokens, marker=\"X\", color=\"red\", markersize=10\n",
        "        )\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "24602363",
      "metadata": {
        "hidden": true,
        "id": "24602363"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4b3d1c5d",
      "metadata": {
        "hidden": true,
        "id": "4b3d1c5d"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.10"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}