CorrectiveRAG.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "authorship_tag": "ABX9TyPMmN6ISseqt+V+zHYkwKrZ",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/MandeepKharb/Youtube/blob/main/CorrectiveRAG.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Corrective-RAG (CRAG)** is a strategy for RAG that incorporates self-reflection / self-grading on retrieved documents.\n",
        "\n",
        "The paper ( https://arxiv.org/abs/2401.15884) follows this general flow:\n",
        "\n",
        "\n",
        "  * If at least one document exceeds the threshold for relevance, then it proceeds to generation.\n",
        "  * If all documents fall below the relevance threshold or if the grader is unsure, then it uses web search to supplement retrieval.\n",
        "  * Before generation, it performs knowledge refinement of the search or retrieved documents.\n",
        "  * This partitions the document into knowledge strips.\n",
        "  * It grades each strip, and filters out irrelevant ones\n",
        "\n",
        "\n",
        "\n",
        "**We will implement some of these ideas from scratch using LangGraph:**\n",
        "\n",
        "* If any documents are irrelevant, we'll supplement retrieval with web search.\n",
        "* We'll skip the knowledge refinement, but this can be added back as a node if desired.\n",
        "* We'll use Tavily Search for web search."
      ],
      "metadata": {
        "id": "LAVnlw641laS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%pip install -U langchain_community tiktoken langchainhub scikit-learn langchain langgraph tavily-python  nomic[local] langchain-nomic langchain_openai langchain_groq"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ns5acg8v2SlL",
        "outputId": "c83c6b55-c559-4cd6-cbf3-81644dbd8e3d"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: langchain_community in /usr/local/lib/python3.10/dist-packages (0.3.14)\n",
            "Requirement already satisfied: tiktoken in /usr/local/lib/python3.10/dist-packages (0.8.0)\n",
            "Requirement already satisfied: langchainhub in /usr/local/lib/python3.10/dist-packages (0.1.21)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (1.6.0)\n",
            "Collecting scikit-learn\n",
            "  Downloading scikit_learn-1.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (18 kB)\n",
            "Requirement already satisfied: langchain in /usr/local/lib/python3.10/dist-packages (0.3.14)\n",
            "Requirement already satisfied: langgraph in /usr/local/lib/python3.10/dist-packages (0.2.61)\n",
            "Requirement already satisfied: tavily-python in /usr/local/lib/python3.10/dist-packages (0.5.0)\n",
            "Requirement already satisfied: langchain-nomic in /usr/local/lib/python3.10/dist-packages (0.1.4)\n",
            "Requirement already satisfied: langchain_openai in /usr/local/lib/python3.10/dist-packages (0.2.14)\n",
            "Requirement already satisfied: langchain_groq in /usr/local/lib/python3.10/dist-packages (0.2.3)\n",
            "Requirement already satisfied: nomic[local] in /usr/local/lib/python3.10/dist-packages (3.3.4)\n",
            "Requirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (6.0.2)\n",
            "Requirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (2.0.36)\n",
            "Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (3.11.11)\n",
            "Requirement already satisfied: dataclasses-json<0.7,>=0.5.7 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (0.6.7)\n",
            "Requirement already satisfied: httpx-sse<0.5.0,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (0.4.0)\n",
            "Requirement already satisfied: langchain-core<0.4.0,>=0.3.29 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (0.3.29)\n",
            "Requirement already satisfied: langsmith<0.3,>=0.1.125 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (0.2.10)\n",
            "Requirement already satisfied: numpy<2,>=1.22.4 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (1.26.4)\n",
            "Requirement already satisfied: pydantic-settings<3.0.0,>=2.4.0 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (2.7.1)\n",
            "Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (2.32.3)\n",
            "Requirement already satisfied: tenacity!=8.4.0,<10,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (9.0.0)\n",
            "Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken) (2024.11.6)\n",
            "Requirement already satisfied: packaging<25,>=23.2 in /usr/local/lib/python3.10/dist-packages (from langchainhub) (24.2)\n",
            "Requirement already satisfied: types-requests<3.0.0.0,>=2.31.0.2 in /usr/local/lib/python3.10/dist-packages (from langchainhub) (2.32.0.20241016)\n",
            "Requirement already satisfied: scipy>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.13.1)\n",
            "Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.4.2)\n",
            "Requirement already satisfied: threadpoolctl>=3.1.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (3.5.0)\n",
            "Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (4.0.3)\n",
            "Requirement already satisfied: langchain-text-splitters<0.4.0,>=0.3.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.3.4)\n",
            "Requirement already satisfied: pydantic<3.0.0,>=2.7.4 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.10.4)\n",
            "Requirement already satisfied: langgraph-checkpoint<3.0.0,>=2.0.4 in /usr/local/lib/python3.10/dist-packages (from langgraph) (2.0.9)\n",
            "Requirement already satisfied: langgraph-sdk<0.2.0,>=0.1.42 in /usr/local/lib/python3.10/dist-packages (from langgraph) (0.1.51)\n",
            "Requirement already satisfied: httpx in /usr/local/lib/python3.10/dist-packages (from tavily-python) (0.28.1)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nomic[local]) (8.1.8)\n",
            "Requirement already satisfied: jsonlines in /usr/local/lib/python3.10/dist-packages (from nomic[local]) (4.0.0)\n",
            "Requirement already satisfied: loguru in /usr/local/lib/python3.10/dist-packages (from nomic[local]) (0.7.3)\n",
            "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from nomic[local]) (13.9.4)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from nomic[local]) (2.2.2)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from nomic[local]) (4.67.1)\n",
            "Requirement already satisfied: pyarrow in /usr/local/lib/python3.10/dist-packages (from nomic[local]) (17.0.0)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.10/dist-packages (from nomic[local]) (10.4.0)\n",
            "Requirement already satisfied: pyjwt in /usr/local/lib/python3.10/dist-packages (from nomic[local]) (2.10.1)\n",
            "Requirement already satisfied: gpt4all<3,>=2.5.0 in /usr/local/lib/python3.10/dist-packages (from nomic[local]) (2.8.2)\n",
            "Requirement already satisfied: openai<2.0.0,>=1.58.1 in /usr/local/lib/python3.10/dist-packages (from langchain_openai) (1.59.3)\n",
            "Requirement already satisfied: groq<1,>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from langchain_groq) (0.14.0)\n",
            "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (2.4.4)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (1.3.2)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (24.3.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (1.5.0)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (6.1.0)\n",
            "Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (0.2.1)\n",
            "Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (1.18.3)\n",
            "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /usr/local/lib/python3.10/dist-packages (from dataclasses-json<0.7,>=0.5.7->langchain_community) (3.25.0)\n",
            "Requirement already satisfied: typing-inspect<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from dataclasses-json<0.7,>=0.5.7->langchain_community) (0.9.0)\n",
            "Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.10/dist-packages (from gpt4all<3,>=2.5.0->nomic[local]) (4.12.2)\n",
            "Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from groq<1,>=0.4.1->langchain_groq) (3.7.1)\n",
            "Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.10/dist-packages (from groq<1,>=0.4.1->langchain_groq) (1.9.0)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from groq<1,>=0.4.1->langchain_groq) (1.3.1)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx->tavily-python) (2024.12.14)\n",
            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx->tavily-python) (1.0.7)\n",
            "Requirement already satisfied: idna in /usr/local/lib/python3.10/dist-packages (from httpx->tavily-python) (3.10)\n",
            "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx->tavily-python) (0.14.0)\n",
            "Requirement already satisfied: jsonpatch<2.0,>=1.33 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.4.0,>=0.3.29->langchain_community) (1.33)\n",
            "Requirement already satisfied: msgpack<2.0.0,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from langgraph-checkpoint<3.0.0,>=2.0.4->langgraph) (1.1.0)\n",
            "Requirement already satisfied: orjson>=3.10.1 in /usr/local/lib/python3.10/dist-packages (from langgraph-sdk<0.2.0,>=0.1.42->langgraph) (3.10.13)\n",
            "Requirement already satisfied: requests-toolbelt<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from langsmith<0.3,>=0.1.125->langchain_community) (1.0.0)\n",
            "Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from openai<2.0.0,>=1.58.1->langchain_openai) (0.8.2)\n",
            "Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3.0.0,>=2.7.4->langchain) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.27.2 in /usr/local/lib/python3.10/dist-packages (from pydantic<3.0.0,>=2.7.4->langchain) (2.27.2)\n",
            "Requirement already satisfied: python-dotenv>=0.21.0 in /usr/local/lib/python3.10/dist-packages (from pydantic-settings<3.0.0,>=2.4.0->langchain_community) (1.0.1)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain_community) (3.4.1)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain_community) (2.3.0)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain_community) (3.1.1)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->nomic[local]) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->nomic[local]) (2024.2)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->nomic[local]) (2024.2)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->nomic[local]) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->nomic[local]) (2.18.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->groq<1,>=0.4.1->langchain_groq) (1.2.2)\n",
            "Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.10/dist-packages (from jsonpatch<2.0,>=1.33->langchain-core<0.4.0,>=0.3.29->langchain_community) (3.0.0)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich->nomic[local]) (0.1.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->nomic[local]) (1.17.0)\n",
            "Requirement already satisfied: mypy-extensions>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain_community) (1.0.0)\n",
            "Downloading scikit_learn-1.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.5 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.5/13.5 MB\u001b[0m \u001b[31m81.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: scikit-learn\n",
            "  Attempting uninstall: scikit-learn\n",
            "    Found existing installation: scikit-learn 1.6.0\n",
            "    Uninstalling scikit-learn-1.6.0:\n",
            "      Successfully uninstalled scikit-learn-1.6.0\n",
            "Successfully installed scikit-learn-1.6.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NFSxP3dT1Sc0",
        "outputId": "898b2647-c926-4f58-d401-1f6ab6f96f0b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "··········\n",
            "··········\n"
          ]
        }
      ],
      "source": [
        "import getpass\n",
        "import os\n",
        "\n",
        "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
        "os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass()\n",
        "os.environ[\"TAVILY_API_KEY\"]=getpass.getpass()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Create Index**"
      ],
      "metadata": {
        "id": "bbGKgT4t5LtO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
        "from langchain_community.document_loaders import WebBaseLoader\n",
        "from langchain_community.vectorstores import SKLearnVectorStore\n",
        "from langchain_nomic.embeddings import NomicEmbeddings  # local\n",
        "\n",
        "\n",
        "# List of URLs to load documents from\n",
        "urls = [\n",
        "    \"https://lilianweng.github.io/posts/2023-06-23-agent/\"\n",
        "]\n",
        "\n",
        "# Load documents from the URLs\n",
        "docs = [WebBaseLoader(url).load() for url in urls]\n",
        "docs_list = [item for sublist in docs for item in sublist]\n",
        "\n",
        "# Initialize a text splitter with specified chunk size and overlap\n",
        "text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(\n",
        "    chunk_size=250, chunk_overlap=0\n",
        ")\n",
        "\n",
        "# Split the documents into chunks\n",
        "doc_splits = text_splitter.split_documents(docs_list)\n",
        "\n",
        "# Embedding\n",
        "\n",
        "embedding=NomicEmbeddings(\n",
        "    model=\"nomic-embed-text-v1.5\",\n",
        "    inference_mode=\"local\",\n",
        ")\n",
        "\n",
        "\n",
        "# Add the document chunks to the \"vector store\"\n",
        "vectorstore = SKLearnVectorStore.from_documents(\n",
        "    documents=doc_splits,\n",
        "    embedding=embedding,\n",
        ")\n",
        "retriever = vectorstore.as_retriever(k=4)"
      ],
      "metadata": {
        "id": "pTCrzgLu5N98"
      },
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import getpass\n",
        "import os\n",
        "\n",
        "if not os.environ.get(\"GROQ_API_KEY\"):\n",
        "  os.environ[\"GROQ_API_KEY\"] = getpass.getpass(\"Enter API key for Groq: \")\n",
        "\n",
        "from langchain_groq import ChatGroq"
      ],
      "metadata": {
        "id": "43zCnHE2Q7lE"
      },
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain.prompts import PromptTemplate\n",
        "from langchain_community.chat_models import ChatOllama\n",
        "from langchain_core.output_parsers import StrOutputParser\n",
        "\n",
        "\n",
        "# LLM\n",
        "llm = ChatGroq(model=\"llama3-8b-8192\",temperature=0)\n",
        "\n",
        "# Prompt\n",
        "prompt = PromptTemplate(\n",
        "    template=\"\"\"You are a teacher grading a quiz. You will be given:\n",
        "    1/ a QUESTION\n",
        "    2/ A FACT provided by the student\n",
        "\n",
        "    You are grading RELEVANCE RECALL:\n",
        "    A score of 1 means that ANY of the statements in the FACT are relevant to the QUESTION.\n",
        "    A score of 0 means that NONE of the statements in the FACT are relevant to the QUESTION.\n",
        "    1 is the highest (best) score. 0 is the lowest score you can give.\n",
        "\n",
        "    Explain your reasoning in a step-by-step manner. Ensure your reasoning and conclusion are correct.\n",
        "\n",
        "    Avoid simply stating the correct answer at the outset.\n",
        "\n",
        "    Question: {question} \\n\n",
        "    Fact: \\n\\n {documents} \\n\\n\n",
        "\n",
        "    Give a binary score 'yes' or 'no' score to indicate whether the document is relevant to the question. \\n\n",
        "    Provide the binary score as a JSON with a single key 'score' and no premable or explanation.\n",
        "    Strictly make sure that value of key 'score' could be either 'yes' or 'no', nothing else\n",
        "    \"\"\",\n",
        "    input_variables=[\"question\", \"documents\"],\n",
        ")\n",
        "\n",
        "retrieval_grader = prompt | llm | StrOutputParser()\n",
        "question = \"What are the types of agent memory?\"\n",
        "docs = retriever.invoke(question)\n",
        "doc_txt = docs[1].page_content\n",
        "print(retrieval_grader.invoke({\"question\": question, \"documents\": doc_txt}))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "z5QE0eU58Rlf",
        "outputId": "47515411-4609-47b5-8d4c-509634cae31b"
      },
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "To grade the relevance of the fact to the question, I will follow the steps:\n",
            "\n",
            "1. Read the question: \"What are the types of agent memory?\"\n",
            "2. Read the fact: \"Memory stream: is a long-term memory module (external database) that records a comprehensive list of agents’ experience in natural language.\"\n",
            "\n",
            "Step-by-step analysis:\n",
            "\n",
            "* The question asks about the types of agent memory.\n",
            "* The fact mentions a specific type of agent memory, which is the \"Memory stream\".\n",
            "* The fact describes the Memory stream as a long-term memory module that records agents' experience in natural language.\n",
            "\n",
            "Conclusion:\n",
            "\n",
            "The fact is relevant to the question because it specifically mentions a type of agent memory, which is the \"Memory stream\". Therefore, I will give a score of:\n",
            "\n",
            "{\"score\": \"yes\"}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# ### Generate\n",
        "\n",
        "from langchain_core.output_parsers import StrOutputParser\n",
        "\n",
        "# Prompt\n",
        "prompt = PromptTemplate(\n",
        "    template=\"\"\"You are an assistant for question-answering tasks.\n",
        "\n",
        "    Use the following documents to answer the question.\n",
        "\n",
        "    If you don't know the answer, just say that you don't know.\n",
        "\n",
        "    Use three sentences maximum and keep the answer concise:\n",
        "    Question: {question}\n",
        "    Documents: {documents}\n",
        "    Answer:\n",
        "    \"\"\",\n",
        "    input_variables=[\"question\", \"documents\"],\n",
        ")\n",
        "\n",
        "\n",
        "\n",
        "# Chain\n",
        "rag_chain = prompt | llm | StrOutputParser()\n",
        "\n",
        "# Run\n",
        "generation = rag_chain.invoke({\"documents\": docs, \"question\": question})\n",
        "print(generation)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Gi1s5V019Qnw",
        "outputId": "65e6bba6-bfc9-4513-e9ca-2d52b3c06f02"
      },
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "According to the documents, there are three types of agent memory mentioned:\n",
            "\n",
            "1. Short-term memory: This is used for in-context learning and is short and finite, restricted by the finite context window length of the Transformer.\n",
            "2. Long-term memory: This provides the agent with the capability to retain and recall (infinite) information over extended periods, often by leveraging an external vector store and fast retrieval.\n",
            "3. Sensory memory: This is used to acquire, store, retain, and later retrieve sensory information, such as visual, auditory, or touch impressions.\n",
            "\n",
            "Note that these types of memory are mentioned in different documents, and there might be other types of memory mentioned in other sources.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "### Search\n",
        "\n",
        "from langchain_community.tools.tavily_search import TavilySearchResults\n",
        "\n",
        "web_search_tool = TavilySearchResults(k=3)"
      ],
      "metadata": {
        "id": "icVqbYMYbEUb"
      },
      "execution_count": 46,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Create the Graph**"
      ],
      "metadata": {
        "id": "IrEDH6Kc9caH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from typing import List\n",
        "from typing_extensions import TypedDict\n",
        "from IPython.display import Image, display\n",
        "from langchain.schema import Document\n",
        "from langgraph.graph import START, END, StateGraph\n",
        "import json\n",
        "\n",
        "\n",
        "class GraphState(TypedDict):\n",
        "    \"\"\"\n",
        "    Represents the state of our graph.\n",
        "\n",
        "    Attributes:\n",
        "        question: question\n",
        "        generation: LLM generation\n",
        "        search: whether to add search\n",
        "        documents: list of documents\n",
        "    \"\"\"\n",
        "\n",
        "    question: str\n",
        "    generation: str\n",
        "    search: str\n",
        "    documents: List[str]\n",
        "    steps: List[str]\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "-8vb3mZ_9fWP"
      },
      "execution_count": 47,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Retrieve** **Node**"
      ],
      "metadata": {
        "id": "OuLsur1syAG8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def retrieve(state):\n",
        "    \"\"\"\n",
        "    Retrieve documents\n",
        "\n",
        "    Args:\n",
        "        state (dict): The current graph state\n",
        "\n",
        "    Returns:\n",
        "        state (dict): New key added to state, documents, that contains retrieved documents\n",
        "    \"\"\"\n",
        "    question = state[\"question\"]\n",
        "    documents = retriever.invoke(question)\n",
        "    steps = state[\"steps\"]\n",
        "    steps.append(\"retrieve_documents\")\n",
        "    return {\"documents\": documents, \"question\": question, \"steps\": steps}"
      ],
      "metadata": {
        "id": "5AVSlMUdwk88"
      },
      "execution_count": 48,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Generate Node**"
      ],
      "metadata": {
        "id": "8UZZ2rz0yMYY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def generate(state):\n",
        "    \"\"\"\n",
        "    Generate answer\n",
        "\n",
        "    Args:\n",
        "        state (dict): The current graph state\n",
        "\n",
        "    Returns:\n",
        "        state (dict): New key added to state, generation, that contains LLM generation\n",
        "    \"\"\"\n",
        "\n",
        "    question = state[\"question\"]\n",
        "    documents = state[\"documents\"]\n",
        "    generation = rag_chain.invoke({\"documents\": documents, \"question\": question})\n",
        "    steps = state[\"steps\"]\n",
        "    steps.append(\"generate_answer\")\n",
        "    return {\n",
        "        \"documents\": documents,\n",
        "        \"question\": question,\n",
        "        \"generation\": generation,\n",
        "        \"steps\": steps,\n",
        "    }"
      ],
      "metadata": {
        "id": "2nG86WmPws4h"
      },
      "execution_count": 49,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Grade Documents**"
      ],
      "metadata": {
        "id": "J5FFCNLAySYj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def grade_documents(state):\n",
        "    \"\"\"\n",
        "    Determines whether the retrieved documents are relevant to the question.\n",
        "\n",
        "    Args:\n",
        "        state (dict): The current graph state\n",
        "\n",
        "    Returns:\n",
        "        state (dict): Updates documents key with only filtered relevant documents\n",
        "    \"\"\"\n",
        "\n",
        "    question = state[\"question\"]\n",
        "    documents = state[\"documents\"]\n",
        "    steps = state[\"steps\"]\n",
        "    steps.append(\"grade_document_retrieval\")\n",
        "    filtered_docs = []\n",
        "    search = \"No\"\n",
        "    for i, d in enumerate(documents):\n",
        "        resp = retrieval_grader.invoke(\n",
        "            {\"question\": question, \"documents\": d.page_content}\n",
        "        )\n",
        "        print(\"here is the Raw response:\", resp)\n",
        "\n",
        "        if '\"score\": \"yes\"' in resp:\n",
        "            filtered_docs.append(d)\n",
        "        elif i == len(documents) - 1 and search == \"No\":\n",
        "            search = \"Yes\"\n",
        "            continue\n",
        "    print('value of search is :',search )\n",
        "    return {\n",
        "        \"documents\": filtered_docs,\n",
        "        \"question\": question,\n",
        "        \"search\": search,\n",
        "        \"steps\": steps,\n",
        "    }\n"
      ],
      "metadata": {
        "id": "Uv1ykkNCw07s"
      },
      "execution_count": 50,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Web Search Node**"
      ],
      "metadata": {
        "id": "PTX_MQXeyX_4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def web_search(state):\n",
        "    \"\"\"\n",
        "    Web search based on the re-phrased question.\n",
        "\n",
        "    Args:\n",
        "        state (dict): The current graph state\n",
        "\n",
        "    Returns:\n",
        "        state (dict): Updates documents key with appended web results\n",
        "    \"\"\"\n",
        "\n",
        "    question = state[\"question\"]\n",
        "    documents = state.get(\"documents\", [])\n",
        "    steps = state[\"steps\"]\n",
        "    steps.append(\"web_search\")\n",
        "    web_results = web_search_tool.invoke({\"query\": question})\n",
        "    documents.extend(\n",
        "        [\n",
        "            Document(page_content=d[\"content\"], metadata={\"url\": d[\"url\"]})\n",
        "            for d in web_results\n",
        "        ]\n",
        "    )\n",
        "    return {\"documents\": documents, \"question\": question, \"steps\": steps}"
      ],
      "metadata": {
        "id": "IKk65h18w9mM"
      },
      "execution_count": 52,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Generation Decision**"
      ],
      "metadata": {
        "id": "lzRWM9EeyiVl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def decide_to_generate(state):\n",
        "    \"\"\"\n",
        "    Determines whether to generate an answer, or re-generate a question.\n",
        "\n",
        "    Args:\n",
        "        state (dict): The current graph state\n",
        "\n",
        "    Returns:\n",
        "        str: Binary decision for next node to call\n",
        "    \"\"\"\n",
        "    search = state[\"search\"]\n",
        "    if search == \"Yes\":\n",
        "        return \"search\"\n",
        "    else:\n",
        "        return \"generate\""
      ],
      "metadata": {
        "id": "Jat9APmmxCl3"
      },
      "execution_count": 53,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Build Graph**"
      ],
      "metadata": {
        "id": "OgQ1EfNkyo_u"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Graph\n",
        "workflow = StateGraph(GraphState)\n",
        "\n",
        "# Define the nodes\n",
        "workflow.add_node(\"retrieve\", retrieve)  # retrieve\n",
        "workflow.add_node(\"grade_documents\", grade_documents)  # grade documents\n",
        "workflow.add_node(\"generate\", generate)  # generatae\n",
        "workflow.add_node(\"web_search\", web_search)  # web search\n",
        "\n",
        "# Build graph\n",
        "workflow.add_edge(START, \"retrieve\")\n",
        "workflow.add_edge(\"retrieve\", \"grade_documents\")\n",
        "workflow.add_conditional_edges(\n",
        "    \"grade_documents\",\n",
        "    decide_to_generate,\n",
        "    {\n",
        "        \"search\": \"web_search\",\n",
        "        \"generate\": \"generate\",\n",
        "    },\n",
        ")\n",
        "workflow.add_edge(\"web_search\", \"generate\")\n",
        "workflow.add_edge(\"generate\", END)\n",
        "\n",
        "custom_graph = workflow.compile()\n",
        "\n",
        "display(Image(custom_graph.get_graph(xray=True).draw_mermaid_png()))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 572
        },
        "id": "IKKOp3ASxIh_",
        "outputId": "16300c34-6bb9-465d-a04f-a54d76a0f3c2"
      },
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMYAAAIrCAIAAAA7vHF/AAAAAXNSR0IArs4c6QAAIABJREFUeJztnXdAE+fjh9/LIAGSsKfsJagoolIHDhRqRVFx4MK9t3VUrbZaW6GOOuqs4sZR3KJ1i3ujVFygIHsmBELITu73x/lL+SriuuQNl/f5KxyX9/0kPLz33t1774vhOA4QCPKgwQ6AoBpIKQTJIKUQJIOUQpAMUgpBMkgpBMkwYAfQK5XlSlGFUlKllohVSnnDuHrCZGF0OmZmweDwGPauLDoTg53oI2DGcF2qJEeenS5+81Rs7chSyDXmPDrXkklrIP9NJmy6SKCsEalqqlT8IrmDO9urGce/NdfE1ECPMBRXil+kuHOab8ZlWNozvZpxrByYsBN9LfmZ0ux0cWmuzM3frG2kDew4dUBlpW6d5OdlSNpH2boHmMHOQj4PLwrvnRNEDHP0C+bAzvI/UFMpXAMOrMxt38vWs5k57Cw6RKMBN4+XM0xo7aMMqLmioFIaNb5lXtbQ+W5WDiaws+iDR1eEkmp1aB9b2EHeQjWl1Ep828Lsyau9YQfRK6mXhGX5sh6jnWAHARRUat/y3KgJzpZ2Db4b/rncO1uB0UBId2vYQah1qfP6MX6nfnZG6BMA4Jse1nKJJueZBHYQCilV/EZWXiCj5MndJ9Kis+W1o2WwU1BIqdvJ/PZRhtJFhQLPmuEeYP70VhXcGBRRKu+lxK4R28mTrZ/qnj59KpfLv+y9arU6LS2N7ERv6dDHNiu9RkeFfyIUUerVY7Gdi54uGSQnJ48aNUoqlX7Z23/99de4uDiyQ72FaYJp1HjBqy/MRgoUUSr7qdgzUE8Xkb+4fSJOrr/47Z+IZzPzN09hNlQN5N5pvRRly9z9zdlm5P975ObmxsfHP336lMfjhYaGLliw4MyZM7///jsAIDw8HACwZMmSqKiotLS0hIQE4nDWtGnTWbNmBQQEAAAqKyvDw8NnzpyZkZFx9epVf39/FxeXixcvAgBat24NADh16pSzszO5mb0DuVf+LiW3zM+CCkpVlit0NOTj119/zcnJmTNnTk1NzcOHD2k0WocOHWJjYxMTE9etW8fhcNzc3AAARUVFcrl83LhxNBrt8OHDM2bMSE5OZrPfdux27NgxcODArVu30ul0c3Pz0tLSwsLCZcuWAQBsbck/n+Ba0/NfSXAcYJBGwVBBqZoqlTmProuSi4qK/P39o6OjAQCxsbEAAGtraxcXFwBAs2bNLC0tid169OgRGRlJvG7SpMmkSZPS0tLatm1LbAkMDJw6daq2TEtLS4FAEBQUpIvABOY8Rk2VimMJ549LBaUkIrW1o0765pGRkbt37165cuW4ceOsrT94YRrDsJSUlMTExDdv3piZmQEABAKB9rchISG6yFYP5jy6RKSGpRQVuucYDdCZOvkgU6dOnT179oULF3r37p2UlPSh3RISEubNm9ekSZM1a9bMmjULAKDRaLS/NTU11UW2emCZ0TQaaPfZqKAU24wuFip1UTKGYUOHDj158mTnzp1XrlxZ+3qS9t6oXC7ftWtX375958yZExQUFBgY+NFidX1ftbJMacaDdvyhglJmPHqNSKWLkokTfnNz80mTJgEAXr58qW11ysvLiX2kUqlcLidO8YizvHdaqXcwNTUVCAT17PD11IjUOupcfgpU6EtZ2DAFRQpdlDx//nwOh9O2bdubN28CAAhvWrRoQafTV69e3bt3b7lc3r9/fx8fn0OHDtnY2IjF4m3bttFotNevX3+ozODg4FOnTsXFxQUFBfF4vE6dOpGbWVajcfUzpTOgPfVAX7p0Kay6ycLClnlmR3HrCGvST5sLCgpu3rx57tw5qVQ6ffr0Ll26AAB4PJ6Dg8PFixdv3LghEol69eoVHBx869atpKSk3Nzc6dOnu7u7Hz16dNiwYUqlcu/evaGhoU2aNNGW6ePjU1VVde7cuUePHllaWpLeec98VK2QaiAOZ6XIeKmzu4r9grneLQxrFDYUzuwoDgjheQVCU4oKBz4AgE8QtyxfXo9SZWVlMTEx72/HcRzHcRqtjj7lzJkziStSOmXcuHF1HiUdHBxKS+u4CN6nT5/vv/++ngLlEo1nU5gj7inSSgEA9v6W22eSs4Vt3ePvVCpVWVkdY4k0Go1Go2Ew6vjXsrCwMDfX+d+mvLxcqazjdFWpVDKZdXwWMzMz7SXW97l/rgIAEPIdzLGd1FEq619x5qNqAxl/DQW1Et/2Y/bkVZDH3VPhIgKBdwsOnUnjF+rk1K9B8PhqZef+drBTUEgpAMC3sQ6HVucBijS7n8fLh9XCMkWTtjzYQailFABgyA9u+3/PhZ1C3xS+kqWlCCOGOcAOAijVl9JSI1Kf3Fw4dIEb7CB6Ive5JO2asM/kRrCDvIVqrRRxH/7b4Y4bZ78Wlujkxp9B8eR61ZOblYbjEzVbKQIcBxf2lWAY1j7KBtYwD52SnV5zO5nfOJjbBuolg/ehrFIEmanVt08LmoTw7N3ZHk2o8IifRKTOfiouyJSq1Xj7KFsre4N7DpbiShFkPKzOfFyd90LSvKMlhgEzHp1jwaDBu7H6WTCYNHGlUiJSS6rV/CK5SKD0bGYeEGLh6MGCHa1ujEIpAlwDcp5LqvgKSbVaVqOWy0geXiKRSF6/ft28eXNyizXn0dUq3JzHMOPS7V3Z9m4GapIWI1JK12RmZi5ZsuTgwYOwg0CGgmd8CLggpRAkg5QiDRqN5u7uDjsFfJBSpKHRaHJzje5e0PsgpciEw0HDSpFSpCIWi2FHgA9SijQwDNPFHAcNDqQUaeA4zufzYaeAD1KKNDAM8/Lygp0CPkgp0sBxPDs7G3YK+CClECSDlCINDMN4PPhDv6GDlCINHMdFIhHsFPBBSpEGhmH1PLRpPCClSAPHcWImICMHKYUgGaQUaWAY5urqCjsFfJBSpIHjeH5+PuwU8EFKIUgGKUUaGIZ5eHjATgEfpBRp4Diek5MDOwV8kFIIkkFKkQaGYZ6enrBTwAcpRRo4jr958wZ2CvggpRAkg5QiDfTQFQFSijTQQ1cESCkEySClyAQ9x4eUIhn0HB9SikzQSAQCpBRpoJEIBEgpBMkgpUgDwzAbGxvYKeCDlCINHMdrL7xutCClSAPdNiZASpEGum1MgJQiDRqNhloppBSZaDQa1EohpcgEwzAHB4NYvgwuaCr9r2Xw4MESiYRYjVgkEhHXERQKxblz52BHgwNqpb6WqKiokpKSoqKi8vJyuVxeVFRUVFRkzPePkVJfy6BBg965tYdhWOfOneElggxS6mthMBj9+/en0+naLW5ubv3794caCiZIKRIYPHiws7Mz8RrDsC5dumh/NEKQUiRAo9EGDRpENFRG3kQhpUgjJibG1dUVNVEAAAou+gsAqKlSCYoVCjnJizjWT+9uE1JSUtq36Pf6X72O7TTnMWydWUyWoSyCSrXrUjVVqqtHykvzZO4BHIlYDTuOPpBWq2qqVL5BnNC+BrE2BKWUqqlSH99U2CXGycLO4BaR1jVPb1dWlcm/Gwn/8j2llNo05/XwxT6YsfYPX96vquLLw4fYw41Bna//3tmKtj3tjdYnAIB/iIWkWl1eqIAbgzp/geI3Uq6V0R3v3oHJogmK5HAzUEcptQpwrY1dKUtblrhKBTcDdS4iSKpVGg11+oVfhkqpqXVnCA7UaaUQBgJSCkEySCkEySClECSDlEKQDFIKQTJIKQTJIKUQJIOUQpAMUgpBMkgpBMkgpT4VsVic+epl/fuoVKrYEdFbtq7TVyhDBCn1qYybMPjs2ZP174NhGJfLY7PZ+gpliFBnJMJXguM4htX3RIBCUd/QNuLtdDp9y6Y9OkjXkDDeVqqqqjKsW+u/k/b9Fre4R8/Qmd+PBwDIZLKNm/6I7h/RM6rTpMnDr6RcIHYePLSXUFhx4uThsG6tBw/tRWwcPTZm2a8L9+5L6NsvPLJXx6ysV2HdWod1a71j52ZihzpLe/HyWVi31qfPHNcm2b1n27fftauqqgQAPE57OGXaqO492g8e2mvFyl8EAj6M7+arMPZWKjFxR58+A/9YvZVOp2s0mkWLvy8pKRo2dLSlpXVa2sNff/tRJpNG9uizdMnKH+ZPC2rRauCAYUwTE+3bHzy4I5PL4n5bK5FKGjVy/XXZ6l+WLSB+VU9pvj6NL1w806tnNLHnxUv/dO4cbmFhmfro/oKFMyLCI6P7DqoWVR09dnD23EkJ2w4ymQ1paKGxK9WkSeC4sVOJ11evXXqS/vjg/mRbWzsAQHi376RSydFjByN79PFv3ITBYNjY2AYGBtV+O53B+GlRnKmpKfFjaIcu2qPn9RtXPlRaz57R69b/XlJS7Ojo9OzZk6KigoXzfwEAbNi4KqpXvxnTfyBKaN267cjRA7KyX/k3bqLfb+WrMHalgoNDtK/v3r2pUqmGxvbWblGr1ebm9U3rExDQTOvTO9RTWreu3239a92ly2djh425cPGMl5dPs2YtSkqKc3PfFBbm1z4mAgCqq0Vf9xH1jbErxWb/J4RQKLCxsV2zemvtHeiM+r4iU3bdPtVfGofD6RrW/dLls4NihqdcvTh2zBRifwDAyBETOnXsWvst9vaOX/TJoGHsStWGy+VVVgodHJxYLFadO3zWM4/1l9azZ/Q/Z0/uS0xQqZTh3XoAADgcLgBALpe5uXl8xYeAj/Ge8b1PcHCIWq0+lXxEu0UqlWpfm7JNP+v8q/7SmgQ08/H2S9y/M7xbD3NzcwCAi4ubg4Pj2XOntLupVCqlUvnVH0vfoFbqPyLCI5NPH9v61/rikiI/X//XrzNv3krZvfMIcekyMLDl5SvnDhzczeXymjZp7uXl8zWlEQ3V+j9XREW9nTkIw7CpU+b8vGTe1OmjekcN0KjV5y+cjoiIHNB/qO4/Opkgpf6DyWSuWrFpe8KGK1fOnz59zMXFrXfUAMb/96UmTphRUcHfl5hgaWE1ZcrsjypVf2kAgPBuPW7cuOLr01i7pWNoWPzydbt2b920+Q9zc07zwJbNmwfr7OPqCurMibBveW7Xoc484346NC2lgsUGId9ZQ8yA+lIIkkFKIUgGKYUgGaQUgmSQUgiSQUohSAYphSAZpBSCZJBSCJJBSiFIBimFIBmkFIJkkFIIkqHO4BZrRxbADWVpHlgwTWhsc8hfAnVaKQYT4xdJP2FHKlP8RmIJe/0c6ijl1cxcWAJ5YQK44BqgUmhcfM3gxqCOUr7BHFyDP75SATsINC7uK2zb04YGeyp96ozqJEhJKscwzMqJZedsCjBKfbQPIRWrK8sVaSmCnmOdHD3gz/BBNaUAAK8ei7Of1qgUuJ4X6NFo1FKpjHjcRZ+YW9Ad3NjBXa3MeLAbKACoqRQsMjMzlyxZcvDgQdhBIEOdvhTCQEBKIUgGKUUaGIZ5eXnBTgEfpBRp4DienZ0NOwV8kFKkgWGYi4sL7BTwQUqRBo7jBQUFsFPABylFGjQazcOjYc/jQwpIKdLQaDQ5OTmwU8AHKUUaqC9FgJQiDdSXIkBKIUgGKUUaGIa5urrCTgEfpBRp4Dien58POwV8kFIIkkFKkYlJrbVAjBakFJnUvxqWkYCUIg0Mw7hcLuwU8EFKkQaO49XV1bBTwAcphSAZpBRpYBhmb28POwV8kFKkgeN4WVkZ7BTwQUohSAYpRRrohgwBUoo00A0ZAqQUgmSQUqSBHroiQEqRBnroigAphSAZpBRpYBjWqFEj2Cngg5QiDRzHCwsLYaeAD1KKNNBIBAKkFGmgkQgESCkEySClSINGo7m7u8NOAR+kFGloNJrc3FzYKeCDlCINDMPQNBtIKTLBcRxNs4GUIhMMw1BfCilFJjiOo74UUopMUF+KAE2l/7UMHz6cz+djGKZQKKqrq62trYnXly5dgh0NDqiV+lo6deokFArLysoqKyvVanV5eXlZWZkxP8mOlPpa+vfv//6Q8zZt2kCKAx+k1NdibW3dvXt3BuO/9VcdHByGDBkCNRRMkFIk0L9/fzc3N+I1juMtW7b09/eHHQoaSCkSsLKy+vbbbzEMAwA4OjqOHDkSdiKYIKXIoV+/fq6urjiOBwUF+fr6wo4DE32vwK5R49UVauqt50kH3G6doi5cuDCw78gqvhJ2HB2AYRY2n2SL/q5LZafXpF2rLMmRWjuy5RK1fipFkIWNE6vgVY1PELdTP1sTdn0HNz0plfFQ/Py+qG2kPcdK3+0igixUCryiRH4xsWjkTx6mnA9apQ+lnt0VZT+RdBnkqOuKEPph7y+vJ6/2oX1AKp13z5VyPPORGPlEJboNcb55gv+h3+pcKX6RXCnT6LoWhD7h2jBzn9d86Lc6V6qKr3T0NNN1LQh9wrNhmnKZmg+cYulcKY0al9aodF0LQs+U5kkAVvev0KVOBMkgpRAkg5RCkAxSCkEySCkEySClECSDlEKQDFIKQTJIKQTJIKUQJIOUQpCMsSg1cFCPNWvjDLM0/aBWq9PT0/RQkbEohVj1x69r1unj36ABKFVQkFfndjSbw2ehkMv1U5EhjgQXCPgbNq5KTb3HYDJbtfrm+vXLf21J9PT0Xv/nimvXL8+dvXjz1rWFhfmrV202YZrsS0xIf5oGAPBv3HTSpFmN/QKIQtRq9d5920+fOS6TSYOCWstlMm35MpksYcemy1fOKRRyVxf3mJjhXcO+rT9SPaUJBPwtW9feu39LpVIFNguaNHGWl5cP8av09LQ9e7c9f5EOAGjRotXoUZP8fP2nzxxryjZduWIjsc/fSfu2/rX+3D+3WCxWVJ8u06fOu5xy/vHjBxwON7xbj+bNW+7avbWgIM/Tw/v773/UfrrHaQ+3J2zMysq0srJuGdRm3NipNja2AICoPl1mzVx482bK3Xs3zc05Ub36jxwxHgDw+8qlKVcvAgDCurUGABzYf8rJ0fnAwd0nTiZVV4t8fBqPGjmxVXAIKX8+g1NKrVb/uGhWhVAwc+aCigr+9oSNLYNae3p6E7+tqRHv2LV51swFMpk0uGWbS5fOyhXy4bHjaDTayZOHFyyccXB/MpvNBgCs/3NF8uljPb7r3aJ58P0Ht6vFb6eP1mg0ixZ/X1JSNGzoaEtL67S0h7/+9qNMJo3s0aeeVB8qTSaTzZ47SSSqmjB+BpvFPvj3ntlzJ+3be5zL4T54eHfhjzO9vXwnTZyl0Wju3LmuVn183Ngfa5dPmTx71MiJf/+99/CR/VdSzs/5fhHb1HTd+t9/+WX+3j3HGAxG6qP7CxbOiAiPjO47qFpUdfTYwdlzJ/21JZH44L+vWDJq5MTBg0devXpx956/GvsFtG0bGjt0THlZaXFx4cIFywAANta2qY/ub0/Y2K3bd9+0aX//wW2pRELGXw8YolKvszIzX71c8vPvXTqHAwDy8nLOnjulUCiIuVAUCsXc2YsDApoRO4eH94iIiCReN27cZPacSelP09q0bpv56mXy6WOxw8aMHTMFANC9e6+0f1OJ3a7fuPIk/fHB/cm2tnYAgPBu30mlkqPHDtajVD2lXbz0T15ezh+rtwS3bAMACAxsOTS297Fjh0aOGL9x02pHR+cNf+4kkvftM/BTPn6P73r36T0AADBx4sxr1y8PGzqmXbuOAIBhQ0bHr1hSVFTg5uaxYeOqqF79Zkz/gXhL69ZtR44e8ODhnY6hYQCAyB59hg0dDQDw8fY788+J+w/vtG0b6uLiZmFhWSEUBAYGEe8qKSkCAET3iWnatLn2OyQFg1NKwC8HADg7uxA/uri4aTQaqVRC/GHYbLbWJ2KWsBs3U5IOJ+bmvjEzMwMACCsEAIAbN64AAAYMGKbdk/b/z3PcvXtTpVINje2t/ZVarTY359QTqZ7S/v03lWPOIXwCADg6Orm5eWRkPi8uKcrLyxk3durnzgrEYrGJFyZMEwCA9u129g4AgKqqypKS4tzcN4WF+afPHK/9xrKyUuIFm21KvKDT6XZ29sT3+T5tvwnlcnlx8T9NnzavbdvQzwpZPwanlKOjM9EL8fP1BwC8ePHU1tbOwsKS+K2p6f8MY9+7L2HX7q39+w2ZMG66oIL/y7IFGlwDACgtK+FwOBY8i/fLFwoFNja2a1Zvrb2Rzqjve6inNHGN2MLSqvYWHs9CwC+vFFYAAOztHD7z038coVAAABg5YkKnjl1rb7e2tn1/Zwadof7AEHEbG9uNf+7ctGXNwkWzmjVr8fPieDs7cpaPNzilvLx82rRuu237n6WlxZVVwlu3ry1etLzOPeVy+YGDu3pG9p02dU7tf1MAgKWFlVgs1h4ua8Pl8iorhQ4OTiwW6xMj1VOana398+fptbdUVAgc7B2JZq9CKHi/NGI2ji+Gw+ECAORymZvbZ0/i+M45spubx4r4Px89fvDzkrkrVi5dvWrz1wTTYogXEaZPm+fi4pZfkGtpYbVxwy6iU/U+MplULpf7/f9JUJWokuh9AwCIjZevnHv/XcHBIWq1+lTyEe0WqVRaf556SmvatHl1tejFi6fEj1lZrwoL8wMDg1xd3e3s7M9fOK36/y45juNENksLK0HFfw/BEX2aT8fFxc3BwfHsuVPa2CqVSqn8+CwMbLZpRYWAyECgUCgAAMEt27Rt2zHz1cvPilEPBtdKqVSqKdNGDhwQ26iRK4Zh1dUisVjM4dTR17GwsPTy8jl2/JC1tU2NWLxn7zYajZad/RoAENYlYl9iwpq1cW/eZPn6NH72/An//7sUEeGRyaePbf1rfXFJkZ+v/+vXmTdvpezeeYQ4XaqTekoL79Zj/4FdS5fNJ8469+1LsLS06tN7IIZhE8bPWB63eOq0Ud27R9FotAsXz0T3iYmIiGzTpt2NtSlJhxODglrfvn3tzD8nPuv7wTBs6pQ5Py+ZN3X6qN5RAzRq9fkLpyMiIgf0H1r/G1s0Dz577tSatXGBzYK4XJ6Vtc0vy+b37RNjamp2//5t/8ZNPitGPRicUgwGo3WrtvsSE7T/31wO98/1Ozw86lg2+KdFcStWLl3260IXF7fJk7/Pyso8evTgxAkzmEzmivgN6zesOJV8xNyc07lTN21vjMlkrlqxaXvChitXzp8+fczFxa131ABGvX0pOp3+odIYDMaqFZs2b1mzZetajUbTPLDl1ClzrKysiXNJNpu9d+/2LVvXWlhY+vkFNHJxI87pCgryDv29d19iQqeO3WIGxu4/sOuzvqKOoWHxy9ft2r110+Y/zM05zQNbNm8e/NF3RUREZmQ+v3DxzJ27N77rHtU1rLu7m+eBA7twHG8R1GrGtB8+K0M96HxOhOd3RfmvZe2jPqPrp1ar6XQ6cbAoKi4cN35wzMDY0aMm6TIm4vPYu+z15FV1T4tgcK2UQqGYPHWEvb1ji+bBTKZJevpjmUzm7e2n63q3J2ys3cHSwuNa7E88qevaqYTBKYVh2LcRPa9cOb9r91YTExNPT58lP//+zgmzLoiJGd6rV7/3t9MwQzyDMWQMTikmkzkoZvigmOF6rteCZ1HnlSfE54L+BREkg5RCkAxSCkEySCkEySClECSDlEKQDFIKQTJIKQTJIKUQJIOUQpCMzpWiMzBTc7qua0HoGScP0w/9SudKWdmbFGWR9kAPwhCoKldIxCpoC37YubKYLHR4pRSV5UqvwA8+U6TzPzaGgaDOluf3FOq6IoR+kNVobp4oad/L5kM76GnxtIJX0hsn+CHf2VnYmbBMUaPVIBEJlJVlihvHSsYt92YwP7ib/pZ4LMuXP7pSmZ9RwzKjS6sNegkQjQbHAMBoX/V01OdUp8Ew2tc9i6Vz7N1NxUKldyCnfe8Ptk8E+lNKi1KGf2j5EUPgxo0b586dW7687ocHdUFNTc2AAQPOnj2rtxq/BAwwTT7pzwZBKUNGo9EUFxc3atRIz/XKZDKxWGxrW8cTww0O1K35HzIzMy0tLfVfL5vNlkqlZWVl+q+adJBS/7Fu3bqHDx+am5tDqd3V1XXWrFkZGRlQaicRdOB7S3l5eWFhYVBQEMQMKpXq2rVr3bp1g5jh60FKAeIZVKFQaG1tDTsIUCgUcrmcy+XCDvLloAMfAAAMHz68tLT0E3bUOSYmJlu3bj106BDsIF8OaqXAq1evWCyWm5sb7CD/cffu3WbNmtU5uYjhY+xKETPpmJp+8L46FHAcF4lEFhYN8lFVoz7wqVSq0NBQQ/OJeIr/5s2bP//8M+wgX4JRK7Vt27b9+/fDTlE3PXv29PPze/z4Mewgn42xH/gQpGOkrVRxcfG6detgp/g42dnZCQkJsFN8Hkaq1A8//BAbGws7xcfx8vKSyWTJycmwg3wG6MCHIBmja6WKioouX74MO8XnUVpaeu3aNdgpPhWjUyomJqZDhw6wU3weDg4OFy9eNPQBVf+PcR34cnJyuFyujc1HxiUaIDiOp6SkdO2q8wkmvx7jUgqhB4zowNelSxexWAw7xVcxd+7c/Px82Ck+grEodfr06fj4+AZ6I1bL6NGjd+7cCTvFR0AHPgTJGEUrtX79egMZDvX1VFdXHzt2DHaK+qC+UklJSTKZzMGB/KXxoMDlcp8/f37ixOctZqRP9H3gUyqVtdfv0gMfWiirfkxMTL5y4TzdoVKpHj9+3KZNG9hB6kbfSlVUVKg+YdVostBoNBiGfYEcdnZ2BquUgUPlA59ara6qqqKkGUKh0GBve1NZKaVSyePxYKfQCVZWVv7+/qdPn4YdpA4ofuD7YtCB74uhbCsllUrV6roXH6cMJSUlNTU1sFO8CzWVUiqVCoWCWHr0HbKysiIjI+/duwcjF8lkZ2cvWLAAdop3oaZSNBqNqr2o2rRv397Hx0coFMIO8j8Y3BKPpECj0YykJzRz5kzYEd4FvlJJSUmnT5+urq729vaOjY0lJrooKSnZvn3748ePWSyWt7f3iBEj/Pz8AADPnj07ePDgs2fPAACNGzceO3asr68vMc9YfHz8Tz/9dPTo0czMzOjo6NGjR8tkskOHDl27dk0gENjb23fr1i0mJobadHmEAAAgAElEQVSoNDc39+jRo69evXJ2dp4yZUrTpk1hfw1fiFwu37lz5+TJk2EH+Q/IB760tLTdu3c3a9Zs+vTp9vb2UqmUOCucO3dudXX1xIkTR48erVKpfvjhh5ycHGLIrEKhGDJkyLBhw0pLS3/++WeZTKYtbfPmzd27d1+wYEFUVJRarV66dOmxY8c6dOgwa9as0NDQgoICbe/q0KFDLVq0mDJlilKpXLZsmQF2cj8RFov16tUrgxpGDLmVKikpAQBERUUFBARohywePHjQ0tIyLi6OwWAAALp27Tpu3Ljz589PnDgxLCxMu5uvr+/ChQufP38eHBxMbImKioqIiCBeX7t27cmTJzNnzuzevfv79U6ZMiU8PJyY1Wn27NmPHz8ODQ3V14cmmQULFhjUTXHISoWEhHC53FWrVk2aNCkkJITY+PDhw/Ly8v79+2t3UyqV5eXlxJPdt2/fPnbsWH5+PvHgee3OaVBQkEKhMDExAQCkpqayWCzCm/fRzrbj7u4OAODz+Tr+oDrE3t7e3t4edor/gKyUtbX16tWrt2/fvnTp0iZNmixYsMDW1lYoFIaEhIwePbr2nsTkdAcPHty3b1+fPn1Gjx5dUVERHx9f+yY0k8mUyWSEUsR8UXVeR6gNjUYjbt3o7CPqg4SEhBYtWhjIjWT43XNXV9dly5alpaX99ttva9asiYuL43A4IpHI1dX1nT3lcnlSUlL37t0nTpxIzFv3zg5qtdrMzIx4zeFwDO3sWnd4eHgcPXrUQJSCf11KoVAQx6yQkJCsrCzi9fPnz1+9eqXdh+i2y2QyuVxOnOIBAEQiETHWQLsbi8Uiul8AgBYtWshksqtXr2p/2yBuBH0Z4eHhI0eOhJ3iLZBbqYyMjPj4+F69epmamqamphK6DBs27MGDB4sXL46Ojra0tExNTVWr1T///LOFhYWHh8epU6esrKxqamr2799Po9GIM0ECpVKpfR0WFpacnLxmzZrMzEwvL6+cnJzHjx9v2LAB0gfVOQEBAbAjvAWyUiYmJq6urklJSTiOBwYGEtdXnJycVq9evWPHjqSkJACAj49PVFQUsf/8+fPXrl37+++/Ozs7jx8/Pjs7++TJk2PGjCF++06LFR8fv2vXrpSUlHPnzjk4OHTq1InCDdXJkydFItHw4cNhB6HQSAS5XE7iUMwGNxLhxYsXy5cvT0xMhB2EQkqRS4NTiuhxGsKMfvC756SgVCqJLrwxw2Qy9Tyuv04oopRCoWhwjQrp7Nq1a9u2bbBTUEUpExMTFosFOwVkgoOD8/LyYKdAfakP0BD7UgaCvi8i6GLRH5FIdPPmzcjISNJLbnDw+XwrK6uP3obSKfpWSheHp/T09Nu3b0dHR5NecoMjLi6uT58+nTt3hpiBCn0pb2/vKVOmwE5hELRo0YIYLwQRNHMLgmSo0ErFxcU9ffoUdgqDQKVSVVZWws1ABaUePXoEa6lPQ0OlUvXq1QtuBiooNWvWrPcHVxknbDbbw8OjoqICYgbUl0KQTINvpcRicVxcHOwUBgSfz4f7wE+DV6qiouLBgwewUxgQBw8ePHz4MMQADV4pOzu7FStWwE5hQAQEBMDtzKC+FIJkGnwr9fr16wa3Yp1Okclkubm5EAM0eKX4fH5aWhrsFAZEVVUV3CkSGrxSXl5ehjCG33CwsbGxtbWFGAD1pRAk0+BbqdTU1Pj4eNgpDIu8vDyI4xwbvFISicSg5i0xBBYtWpSZmQmrdvhzInwZs2fPTklJIWa7w3E8ODgYwzA7O7tz587BjgYfPz+/2g9e65mGqtTw4cOfPXsmEAiIGYKIceLa6YSMnJ9++gli7Q31wNeyZct3ZkN0cHBAp34EQqEQ4mKWDVUpAMCIESOsra21P7Zq1Uo7qYuRs2fPHohLYTVgpYKCgpo1a0a8dnZ2HjFiBOxEhoKjo6N2UiT901D7UgSjRo169uwZn88PCQnx8fGBHcdQGDx4MMTaG3ArBQBo3rx5kyZNHB0dhwwZAjuLASGRSCCOQP/I1fOyfPmjlMqSHKm02kBns8RxHMdxYspNA8TBna3R4N6BnJZhlnqr9MyZM/fu3Vu2bJneaqxNfQe+N88kd/8RtOhs06KzjSkH5gOsDRgc8IvkFcXyw+sLBs500U+dlpaWEGeI+GAr9fyuKCO1JjzWSe+RqMmbdPGLe8JBc6j/2EXdxwu5RJP5SIx8IhHPQI5HM+6/1/TRxVEqlcTcuFCoW6mibClGQ/OWkIylncmbZ/p40ODZs2ezZ8/WQ0V1UrdSogqVowf8Kfooho0jm6aXf1Q2m81ms/VQUZ3U3T2XS9QKWZ2/QXwFGCjL18fX6u/vv3HjRj1UVCcGeu6N+Bo0Gg3ER/mQUhQkNzcX4mINSCkKwmQyibWZoICUoiAuLi4HDhyAVTtSioLgOF57MVU9g5SiIHw+v2/fvrBqR0pREBqNxmQyodUOq2KE7rCxsUlOToZVO1KKgqC+FIJkhEKhdgVD/YOUoiB0Oh3ifLhIKQpiYWFh7E/IvHqdEdat9Z07N2AHqY8GEZIAx/GqqipYtRuEUghyqaqq6tevH6zakVL/AzWmRqLT6Y6OjrBqJ+c5vvkLZxQU5O3f9/b4nbh/p6eHd4cOb9dbGjl6QEBAswU/LAUAnDx1JOlwIp9f5ujo3K3rd4NihmsH3l+5emHrtvUlJUU+Po0njp/RvHnL+is9cHD3iZNJ1dUiH5/Go0ZObBUcAgAoLinavHlN6qN7JiYsP1//MWOm+DduAgBIT0/bl5iQ/jQNAODfuOmkSbMa+wUAAK5eu/TLsgW//rL678P7Xr58NmTwyDGjJ8tksn2JCSkpF8r5ZQ4OTt9G9Bw2dDRR6ZucrENJezMynru4uM2cPj8wMIiUL5BcuFzu/v37YdVOTivVpXN4UVHBmzdZxI/nzief/uc48To7+3VeXk6XTuEAgN17tm3b/mfXsG/nzf25S+fwv5P2/rF2ubaQnDdZA/oPHTVyYmlp8Zx5k58/T6+nxtRH97cnbGzePHj2rB8dHZykEgkAQCDgT58xRlRdNW3q3IkTZiiVypmzxhGpSkqK5Ar58NhxI0dMKCkpWrBwRu0rN+s3rOgVGb1yxcaoXv3VavWPi2YlHU7s2LHrD3N/7typW35BrnaFu8T9O1oGtZk1c4FCoVj002yIUw/Ug0ajgbjeFTmtVIcOXRhr427dvubp6f3vv48KC/OLiwtLS0scHByvXb/EMee0avUNn1++/8DOxYuWd+7UjXiXjY3d2nXx06bOJX4cM3pyu3YdAQAR4ZGjxgxI2LFpzR9bP1RjSUkRACC6T0zTps0jIt4u7rgvMcHK0vqPVVuIx7cjwiNjR/Q9/c/x6VPnhof30O7WuHGT2XMmpT9Na9O6LbEluu+g7t3fLr1yJeXC47SH8+b+FNmjz/v1zpw+n9jT3c1zyrRRjx7f79SxKynfIYmIRKJhw4ZdvnwZSu3kKMXj8oJbtrl162rssDFnz58KatGqQig4e+7UqJETrl671CG0C5PJTE29p1KplsctXh63mHgX0XHhl5e9U5qtrV1oh7BLl8+qVKoPPdvf9ptQLpcXF//T9Gnz2rYNJTbeu3errLw0sldH7W5KpbK8rJSYMOjGzZSkw4m5uW/MzMwAAMIKgXa34OD/ZhG6/+A2i8Xq/m3di/vweBbECw8PbwAAUbihQaPRLCwsYNVO2pwInTuHr1r9a15ezrVrl36Yt6RCwE86ktgxNCwvL2fyxFkAAEEFHwAQt3ydvZ1D7Tc6O7u8ycl6pzQ7O3u1Wi2TyTgcTp3V2djYbvxz56YtaxYumtWsWYufF8fb2dlXCAXt2nWcMG567T3NzTkAgL37Enbt3tq/35AJ46YLKvi/LFugwTXafcxMzbSvhRUCWxu7j67lSjzfrNYY4kPYPB7v2LFjsGonTakOHbqsWRsXv2KJqalZx9AwqUy6fcfGNeviiKMeAIDL5RF7url5fLQ0obCCzWbXfwnYzc1jRfyfjx4/+HnJ3BUrl65etZnL5VVVVb5fvlwuP3BwV8/IvtOmzgEAlNXbtHA43AqhoJ4dGgRSqdTUFM4zTqRdRLDgWQS3bPPy5bPIHn0YDAaXww3r8u3z5+nEUQ8A0LJlGwzDjp/4W/sWqVRaZ1EymezuvZtBQa3rXwNdoVAAAIJbtmnbtmPmq5fE8evp038zMl+8U4VMJpXL5X5+AcTGKlEl0Yets9iWLdtIpdLLV85rtzSIJeNrU1lZCXFVPjInA+rcOfxh6r1ePd9eZOvde8C588nEuR4AwKWRa7/owUePHfxx8fehHboIBPwTJ5Pi49b7+foTOyTs3FQhFEgkNefOJ4tEVaNGTqynrhcvn/2ybH7fPjGmpmb3798mrhSMHDHh7t2b836YGjMw1srK+v7922qN+rdlf1hYWHp5+Rw7fsja2qZGLN6zdxuNRsvOfl1nyRHhkSdOJv2+YsnLl898vP2y37xOfXRv21Zo5+RfAIZhsJookpUK7dDl7t2bjo5vH3sP8G8a3LINcdQjmDpltr29w/Hjfz94cMfGxrZjaJidrT3xKzc3j9AOXfYlJlRWChs3brJm9dbG/9+o1IkJ08TdzfPAgV04jrcIajVj2g8AgEbOLhv/3Lnlr3X7D+zEMMzX1z+67yBi/58Wxa1YuXTZrwtdXNwmT/4+Kyvz6NGDEyfMeL9kFov1x+qt27dvuHjpn9Nnjjk6Ood1+bZhNVQWFhanT5+GVXvd02zcP1chl4GgMOu63oL4QuQSzYmNOeOWe+m6IhzH5XI5rAeODXoWvO0JG08lH3l/O49rsT/xJIxEDQOBQBAbGwtrvm6DViomZnivXnXc/qRh6NbkR7C01N8Mae9g0EpZ8CwseNAu2TVcbG1tDx06BKt29O9OQdRqNcRFUJBSFKSkpGT8+PGwakdKURAajVZ7kQF91w6rYoTucHJy2r17N6zakVIURKPRoOf4EGSSkZExbtw4WLUjpSiISqWCOFcnUoqCBAYGJiQkwKodKUVBcBz/0NAdPVC3UkwTmgkb2UY2NGDloI91OG7cuDFnzhw9VFQndXtjbkEXFMv1HobiiMoVGo0+nhNUqVQQx57XPbhFUKy4+09FpwHQHi+kJNlPxJIqebteNrCD6Ja6WykbJxOeNSMtpULveSiLWoXfOlmqH5+kUml1dbUeKqqTD3aYOkbb4hr84XmBUg6to0cZBIXyI2tzxvziqZ/q/v777z179uinrvepb3BLp342j64IT23NAzgw4xjoMBgcxzU4TjfUJR65tszsJ9W+LbkjFrvr7YyHRqNBnBPhI6uGAgBwHFQLVRKRgQ6+/vfff1NSUmbNmgU7SN3QGJitM8tQhdcJH297MAzwrBk8awNtpV7lK2SgxNED2sViA0ShUNBoNFiLsBvTv4/RMHv27IcPH8KqvcErRafTiTkOEFrEYjEV5kSABdyFwgwTiIOlqNBKmZiYODg4fMKORoRaDXPyjwavFAAgPz8fdgQDQiQShYeHQwzQ4JVisVg2NhS/xfFZCAQCT089XVOtEyoo9ebNG9gpDAhPT8+dO3dCDNDgleJwOKh7XhuRSMTn8yEGaPBK8Xi8+qehMjZ27tx59uxZiAGooFRxcbFSqYQdxFCQSCS+vr4QAzT461IAgJCQkIqKCnQpgeDHH3+EG6DBt1LELa28vDzYKQwFgQDyRKNUUMrd3T03Nxd2CoMgIyNj+vTpn7CjDqGCUgEBASKRCHYKgyAvLy8kJOQTdtQhHx8vZfikpaVt2LBhx44dsIMgAEVaqWbNmj19+hR2CoMgOzsb4mwIBFRQisFgdO3aNSMjA3YQyIhEorFjx0J8dJ2ACkoRPfQbNxrAep46JT8/f+zYsbBTUEWp9u3b37p1C3YKyDRt2jQ2NhZ2Cqoo1bx5cxMTk8rKSthBYHLhwgW5HP4z4hRRivgfPXnSeCdDT01NPXLkiHYJVohQR6nevXufOnUKdgpoVFRUTJ48GXYKQJF7fAQeHh4BAQFpaWlBQYa43rCuiYiIgB3hLdRppQAA0dHRmzdvhp0CAs+fP4e1vMf7UEqpVq1a4Tj+6NEj2EH0TVxcnIfHxxfO1A9UuCFTm/T09J07d65duxZ2EP0hkUjKysoMRylKtVLENJVcLvfMmTOwg+gPMzMzw/GJgkoRY9BWr14NO4WeSEtLgzgfdZ1QUCk2mz116tT4+HjYQfTB4cOH582bBzvF/0C1vpSWyZMnjx49GvrgISOEgq0UwcqVKyk/gur48ePQh7K8D2WV4nK5AwcOnD9/PuwgumLdunU1NTXQh7K8D2UPfAQrVqzw9PSMiYmBHYRk5HJ5cXGxQZ3oaaFsK0Uwf/78O3fuZGVlwQ5CMmKx2MXFBXaKuqF4K0VMQPXNN988ePAAdhDSiIuLa9y4cf/+/WEHqRuKt1LE9LqJiYmLFy+GHYQcMjMzW7ZsabA+GUUrRXD+/Plr167FxcXBDkJ9qN9KEXTv3r1Zs2Z79+6FHeSrCAsLM4Rxm/VjLEoBAIYOHVpUVHTkyBHYQb6Q7du3Hzp0yBDGbdaPESkFAFiwYEFOTk6DePBh2rRp72wZP358g5hKxLiUAgDMnTt3z549qampAID+/fsHBwdPmjQJdqh3ycrKysvL69SpE/Hjhg0bLl68CDvUp0KdgcKfzrZt28aPH19SUlJcXEyj0UpLS8vKyuzt7WHn+o8nT57w+XyFQtGhQ4dNmzaFh4cHBATADvWpGF0rRVBcXFxcXEy8FggEaWlpsBP9D7du3SK64XK5fNKkSQ3IJyNVqmfPniUlJdofJRKJQT2pXFlZ+fr1a+1skSqVqlWrVrBDfQZGp1RUVFRZWdk7G1++fGk41+fS09Orqqpqb8EwrF27dvASfR5Gp1RycvKYMWN8fHysrKy0GlVXVxvOse/u3bvEdFk4jmMY5uTk1KZNmwY0pMJYrp6/z40bN86ePfvixYvi4mKFQjF27NipU6fCDgUAAIMHD87IyLCysrKxsenSpUt4eHjjxo1hh/oMqKPUs7ui4myZWoWLBJ8xu7BarZLUSETV1TQazcnJSZcBP5X8/HyWiQmXyzX9zBW8LOyYZhy6TwuOoyfMQVRUUArXgCPrCxr5mpty6NaOLLW6wX+iLwPXgPICWVm+1N3frEUnaIunUUGpI+sKm4ZaufiiVfnecutEmaMHK6gzHKsafPf81imBT0se8qk2Hfra52dKS3PgDEtv8Eq9uC9qhHx6D7tG7Ff/iqFU3bCVqhGprR1N2OZ02EEMDlsXtqQazkKPDVsplUIjFqpgpzBEaHRQVQ5nXZ2GrRTCAEFKIUgGKYUgGaQUgmSQUgiSQUohSAYphSAZpBSCZJBSCJJBSiFIBimFIBmkFIJkkFIGQUlJcXFJEewU5ICUgk9hUcHQ2N4ZGc9hByEHY1eqsKhAD0Ol669CrVJRYLi2FqObE0GpVO7cteXS5bNSqaR58+DMzBfDY8f16T0AAFBcUrR585rUR/dMTFh+vv5jxkzxb9wEABDVp8usmQtv3ky5e++muTknqlf/kSPGE6XJZLKEHZsuXzmnUMhdXdxjYoZ3DfsWAHD12qVfli349ZfVfx/e9/LlsyGDR8YOG7t33/YrV86XlZfa2Nh+G9Fz1MiJdDq9uKRo5OgBAIBfli34BYDu3Xst+GFpPWEMH6NTauu29adOHRk3dqqtrf2WrWvlclmP73oDAAQC/vQZYxo1cp02dS6GYRcunJk5a9zWzfs8Pb0BAL+vWDJq5MTBg0devXpx956/GvsFtG0bqtFoFi3+vqSkaNjQ0ZaW1mlpD3/97UeZTBrZow9R1/oNK8aNmTpm9GSXRm50Oj019V679p2cnVxev85I3L+Ty+XFDIy1sbZd9ONvy+MWjx41qWVQaysr64+GMXCMSymNRnP69LGekX0HxQwnjkfL4xanP01rFRyyLzHBytL6j1VbGAwGACAiPDJ2RN/T/xyfPnUuACCyR59hQ0cDAHy8/c78c+L+wztt24Zev3HlSfrjg/uTbW3tAADh3b6TSiVHjx3UKhXdd1D37r20tW/etEc700FRccH1G1diBsaamJj4+foDANzcPAID3y5OWX8YA8e4lJJIJAqFolEjV+JH4kV1tQgAcO/erbLy0sheHbU7K5XK8rJS4jWbbUq8oNPpdnb2An45AODu3ZsqlWpobG/tW9Rqtbk5R/tjcPD/LDciFFbs3bf9wcO7RI1cDvdDOesPY+AYl1JmZmYcc056etrAAcMAAC9ePAUAeHv5AgAqhIJ27TpOGDe99v61/dDCoDPUGjUAQCgU2NjYrlm9tfZv6Yz/vlIz0/8e3amoEEyYNMzU1GzM6MnOzi47d27OL8j9UM5PD2OAGJdSNBptyJBR2xM2/rZ8ka2t/clTh/v3G+Lq6g4A4HJ5VVWVbm6fsd4Bl8urrBQ6ODh9yvyZp5KPCoUVmzbsdnBwBADY2zvWo9QXhDEcjO4iQt8+MW1atxUKK8Ti6kU//jZt6hxie3BwyNOn/2ZkvtDuKZVK6y8qODhErVafSv5vPtl63iISVVpaWhE+AQCqRJXaCwcsFhsAQBxMvziM4WBcrRQA4NflP/J4Fu3adQIAYAArLS0h/swjR0y4e/fmvB+mxgyMtbKyvn//tlqj/m3ZH/UUFREemXz62Na/1heXFPn5+r9+nXnzVsrunUfqXCooKKj18RNJO3dtadq0xY0bV+7du6XRaKqqKi0sLO3tHZydGiUdSWSbmopEVf2iB39BGMOBvnTpUtgZvhy5RJPxsDrgG8tPf4tQKDh95tjlK+ev37hyJeXC8RN/Ozo4e3v78bi8Du075+a9uXjxzIOHd8zNOT0j+3p4eAEADh7a7evr36Z1W6KE06ePmZtzuoZ1p9PpXTpHiMWiq1cvXr9xpUYi7vFdn8DAIBqNlpObfe3apei+MRYWb7O5u3viuObEycM3rl92buQ6d85P6emPpVJJUFBrDMOaNGl+/8HtKynni0uKQjuEOTs1+lCYT0QiUhVnSZq2433mN0oCDXuajSq+8uSWougZ7p/+FrVaTae/fTpZVC1asHAGg8H4c12CzjLCobxAlnqBP/B7CEsXGd2B7481y7OyMtu162RpaZWXn5Od/apnz2jYoSiF0SkVEtK+rKzk6LEDSqXSyanRiOHjiQsKCLIwOqW6dA7v0jkcdgoqY3QXERC6BimFIBmkFIJkkFIIkkFKIUgGKYUgGaQUgmSQUgiSQUohSKaBK4VjLFM0Q3UdYDSMycKgVN2wleJY0QUlhr7IPRTEQiWsf7aGrRSdgTm6s9HU5+9TXaF08ICz3lXDVgoA0KKz5f1z5Z+woxGhUuBPblQEh33GyEQSadhD8Ahe3Kt+/aSmS4wj7CAGQXWF8vrRksgxTjxrOMNMqKAUsb7jq0dipULj6GkmhbR2CnSYLKzwlYRjyeg2xJ5rBW3YEkWUAgAo5ZryQoVIoFQqNFAClJWVJScnjx07FkrtAAC2Gd3GycTa0QRWAALqDMFjsmjOXmxnL2hrsGZmlpYmPQjsMBtWAAOhwXfPEYYGUgpBMkgpMjE1NYUdAT5IKdLAMMzc3Bx2CvggpUgDx3E+nw87BXyQUmSCWimkFMnU1NTAjgAfpBSCZJBSpIFhWKNGjWCngA9SijRwHC8sLISdAj5IKTKxtraGHQE+SCkyqaiogB0BPkgpBMkgpUgDwzBPT0/YKeCDlCINHMffvHkDOwV8kFIIkkFKkQaNRkMHPqQUmWg0GnTgQ0ohyAcpRRo0Gs3d/TNmYKcqSCnS0Gg0ubkfXGnIeEBKIUgGKUUaNBrNyckJdgr4IKVIQ6PRFBcXw04BH6QUgmSQUmSCxp4jpUgGjT1HSpEJhmG2trawU8AHKUUa6Dk+AqQUgmSQUqSBRiIQIKVIA41EIEBKkQaGYTwehBXPDQ2kFGngOC4SiWCngA9SCkEySCnSwDDMysoKdgr4IKVIA8dxoVAIOwV8kFKkga6eEyClSAPH8erqatgp4IOUIhO5HC27RaHVGWAxdOjQly9fYhhGXO2k0+nEV5qamgo7GhxQK/W1TJ482crKCsMwDMPodDrRqfLx8YGdCxpIqa+lY8eO3t7etbeYmJj0798fXiLIIKVIYPjw4bVvxbi7u/fr1w9qIpggpUiAaKiILhSdTu/Xrx+DQZ31nj4XpBQ5xMbGEgPPXVxcjLmJQkqRRufOnX19fWk02oABA4hOutFipBcRNGq8IFNaXaWSiFQaNZCKSVhotLi4+NGjR5GRkcQFha8Bo2EMJjDjMsx4dEtbpoM7tEUGvwCjU+rZHVHGI3FxtsTRi6dS4jQTOsOEqdEY1peAYRiuVmtUarVCTWcAYYnEK5Dj15Lr5t8AVtIyIqXSrlXeThbYefJMeaYc2wbwt9GiUmhEZTVApVRK5J2ibZ3grYz6KRiFUvxCxbm9pUxzlp2XNY3+tUcliEgq5fxsgbMnO3yoHewsH4T6Sr18WH3nTIVrkDPDhCLnIjUCWemr8uE/ujFZhviJKK5UzjPJ3fNVjgH2sIOQjEKqyrpbMGG5N50JO8p7UFmpJzernt6RODejmk9aXlzJGfebF5NlWIdyQ2w5SaH4jSztuojCPgEAvNu5JMYb3Lx71GylVApwZEOhYxNH2EF0jkQoNaFJwwcbUG+dmq3UtWPlJlwz2Cn0gZmVacErWUmODHaQ/6CgUuJKVXa62NrVWJ7StPO2vn7cgKb3oKBSqZcrHfwM8bECviB/7k/fPH5ygdxiza3YNCazIFNKbrFfDAWVenGvimPdkC6Ofz2YiUnmY0N5koJqShW8kppbsWgMwzqv1jU8O/M3zySwU7yFaiPFCl9LuXZcHRX+Ojv1n4ubi0oyuRxrH8/WPSIm87i2hUUZGxPGjx2+9p8Lm4tKMq0snXp+O61ZQCfiLeIa4cl/1lufvMMAAARJSURBVD57eZ3JYHl7ttJRMCabzrVh84sUts4mOqri06FaK1WcI6cxdfKhXmU92L53hoO9Z0zfRZ3aD83Oebx111SFQgYAUCrliX8v6tR+8OQxW6wsHQ8c/qmmphIAoFQp/to9/dmLa53aD+3ZfVqFsEgXwQgUMo2Ir9Rd+Z8O1VopiUhlZaWTEXAnzvzRtnV0dK+5xI9+Pt+s+nNQxuu71pZOAIC+PecEBUYAACIjpqzbMjIr53HzpmG37h4uLnk1YeQGP58QAICHa+DKPwfpIhsAgMZk1IhUOir8s6CcUmKVHYv8D1UhLC4tf8OvyL/78ETt7ZVVpYRSJsy3JwRWlk4AAFF1OQDg6YtrTg4+hE8AABpNh6M9GSy6uAoppQN01C2vFgsAABFh45o3Cau9ncu1ragorL2FQWcCADQaNQCgsqqkkVNj3SQyXKimlCmXoZKrmGyS2wNTNpfoM9nbeXz6uzjmVuIaPc3lolaoORYs/dRVP1TrnptbMJRyEgaSv4OdrZulheODR8lyxdsrimq1SqX6SHe4kVPj/MLnZeX6uLOrUanMLQyigaCaUo7uLI2SfKUwDOsT+b2omr/hr7G37h25cefvP/8ae/v+kfrfFdZxBIbRNu+cdOX6noePzxw7vYr0YFpMTDCejUEMnqKaUi4+ptXlYl2UHNiky5jYNXQ689Q/ay9d3Wll5ejl0bL+t9jauIwfsd6SZ3/+yvaLV3c6O/jqIhgAQClTVQvkhnBRipqDW7bOz/ILdaMxqPbfUg+CPJGNtarLQIMY4mIQR19yadLWslIg4zl8cHDLhSvbr9859P52Fyf/guKXdb5l+vgEB3vSpsn/5+Lm2/ePvr/dlM2Vyuq+VTdr8h5ba5cPFahRKHyCDGXkBQVbqZoq1YGV+b6hbh/aQSIRyeR1HBwx7IPfhgXPnk4n7d+vRlIll9exJhaOgw89VVpPgJoKmaRcOHDWB4XTMxRUCgCQcrhcKGQYyZCp3NSi70bYO7gZxBUECnbPCTpG26nEhnJnXqdIhFJ3f1PD8YmySjEYoPMA27xHOrxNawgoJMryLEGXAYY13pCaSgEAHNxYwWG8wqelsIPokNd3Coct+GCXERbU7EtpyXshvXW20ol6j4ZKVK/vFEz83Zu8cwbSoGwrReAWYNo6jPPmXoFKB3dpYFFTIS16VjIh3ssAfaJ+K0UgLFWc21tGZ5vYelg36DHENUKZ4E2FW2NTQ+s/1cYolCJIv1l16xTfxo3H5rG5dg3pKT+VXC0qkwCVQq1QdOxr6+iBJgMyJF7crc54XF2QKXH05qlUGjqTwWAxDe1LwACmUavVSpVaoaYzMFG51CuQ49uS4+rXAJ78MTqlCHAcFGRKxJWqGpFarcJJmViRRDAMY5gAcx7DnMewsGPauxrQZaePYqRKIXQHxc/4EPoHKYUgGaQUgmSQUgiSQUohSAYphSCZ/wOZGSWHxu797AAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import uuid\n",
        "\n",
        "\n",
        "def predict_custom_agent_local_answer(example: dict):\n",
        "    config = {\"configurable\": {\"thread_id\": str(uuid.uuid4())}}\n",
        "    state_dict = custom_graph.invoke(\n",
        "        {\"question\": example[\"input\"], \"steps\": []}, config\n",
        "    )\n",
        "    return {\"response\": state_dict[\"generation\"], \"steps\": state_dict[\"steps\"]}\n",
        "\n",
        "\n",
        "example = {\"input\": \"who is sachin tendulkar ?\"}\n",
        "response = predict_custom_agent_local_answer(example)\n",
        "response"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2urAsI6Y-D6O",
        "outputId": "ece21ba0-10bd-48fc-b8f6-dda2ce408eec"
      },
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "here is the Raw response: To grade the relevance of the fact to the question, I will follow the steps:\n",
            "\n",
            "1. Read the question: \"Who is Sachin Tendulkar?\"\n",
            "2. Read the fact: The fact is about LLM-based evaluation, ChemCrow, GPT-4, and scientific discovery. There is no mention of Sachin Tendulkar.\n",
            "\n",
            "Conclusion: None of the statements in the fact are relevant to the question. The fact is about AI, scientific discovery, and evaluation, whereas the question is about a person, Sachin Tendulkar.\n",
            "\n",
            "JSON score:\n",
            "{\n",
            "\"score\": \"no\"\n",
            "}\n",
            "here is the Raw response: Let's break down the fact and question step by step:\n",
            "\n",
            "1. The question is \"who is sachin tendulkar?\"\n",
            "2. The fact provides two unrelated pieces of information:\n",
            "\t* A picture of a sea otter using a rock to crack open a seashell, which is about animal behavior and tool use.\n",
            "\t* A description of a neuro-symbolic architecture called MRKL, which is about artificial intelligence and computer science.\n",
            "\n",
            "Neither of these pieces of information has any relevance to the question about Sachin Tendulkar, who is a cricketer.\n",
            "\n",
            "Therefore, I conclude that the fact is not relevant to the question.\n",
            "\n",
            "Here is the JSON score:\n",
            "\n",
            "{\n",
            "\"score\": \"no\"\n",
            "}\n",
            "here is the Raw response: Let's break down the fact and question step by step:\n",
            "\n",
            "1. Question: Who is Sachin Tendulkar?\n",
            "2. Fact: To avoid overfitting, CoH adds a regularization term to maximize the log-likelihood of the pre-training dataset. To avoid shortcutting and copying (because there are many common words in feedback sequences), they randomly mask 0% - 5% of past tokens during training. The training dataset in their experiments is a combination of WebGPT comparisons, summarization from human feedback and human preference dataset.\n",
            "\n",
            "Step 1: Identify the main topic of the fact. The fact appears to be discussing a machine learning technique called CoH, its regularization term, and the training dataset used in experiments.\n",
            "\n",
            "Step 2: Compare the main topic of the fact to the question. The question is asking about Sachin Tendulkar, who is a cricketer. The fact has no mention of Sachin Tendulkar or cricket.\n",
            "\n",
            "Conclusion: None of the statements in the fact are relevant to the question. Therefore, the score is:\n",
            "\n",
            "{\"score\": \"no\"}\n",
            "here is the Raw response: Let's break down the process step by step:\n",
            "\n",
            "1. I read the question: \"Who is Sachin Tendulkar?\"\n",
            "2. I read the fact: \"FAISS (Facebook AI Similarity Search): ... ScaNN (Scalable Nearest Neighbors): ...\"\n",
            "\n",
            "Now, I need to determine if any part of the fact is relevant to the question.\n",
            "\n",
            "Upon reading the fact, I don't see any mention of Sachin Tendulkar, cricket, or any other information that would be related to the question. The fact appears to be discussing AI-related concepts, specifically Facebook AI Similarity Search and Scalable Nearest Neighbors.\n",
            "\n",
            "Therefore, I conclude that none of the statements in the fact are relevant to the question.\n",
            "\n",
            "The binary score is:\n",
            "\n",
            "{\"score\": \"no\"}\n",
            "value of search is : Yes\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'response': 'Sachin Ramesh Tendulkar is a former cricketer and captain of the Indian team, considered one of the greatest batsmen of all time. He has set many records in his career, including being the only player to score over 30,000 runs in international cricket. Tendulkar is also known as \"The Little Master\" and \"God of Cricket\".',\n",
              " 'steps': ['retrieve_documents',\n",
              "  'grade_document_retrieval',\n",
              "  'web_search',\n",
              "  'generate_answer']}"
            ]
          },
          "metadata": {},
          "execution_count": 63
        }
      ]
    }
  ]
}