langchain_testing.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1ba6a7a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.llms import Ollama\n",
    "llm = Ollama(model=\"llama2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5b89c289",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nLangsmith is a tool that can help with testing in several ways:\\n\\n1. Automated Testing: Langsmith provides a built-in test framework that allows you to write and run automated tests for your code. This can save time and effort compared to writing and running tests by hand.\\n2. Code Coverage Analysis: Langsmith can analyze the coverage of your tests, providing insights into which parts of your code are not being tested enough. This can help identify areas where you need to write more tests.\\n3. Test Data Generation: Langsmith can generate test data automatically, based on the structure of your code. This can be especially useful for complex systems with many interdependent components.\\n4. Code Refactoring: Langsmith can help you refactor your code by identifying areas where the tests are not covering enough code. This can help you to write more efficient and effective tests.\\n5. Integration Testing: Langsmith provides integration testing features that allow you to test how different components of your system interact with each other. This can help ensure that your system works as expected in a real-world environment.\\n6. Mocking and Stubbing: Langsmith provides tools for mocking and stubbing, which can help you isolate specific parts of your system when testing them. This can make it easier to write and run tests, and can help ensure that your tests are more reliable.\\n7. Test Automation Framework: Langsmith provides a test automation framework that allows you to organize and run your tests in an efficient manner. This can help you to write and run more effective tests, and can make it easier to maintain and update your tests over time.\\n8. Continuous Integration and Deployment: Langsmith can be integrated with continuous integration and deployment (CI/CD) tools, such as Jenkins or Travis CI, to provide automated testing and deployment capabilities. This can help you to quickly and easily test and deploy your code changes, and can ensure that your system is always running smoothly.\\n9. Testing of Legacy Code: Langsmith can be used to test legacy code that may not have been designed with testing in mind. This can help identify issues and bugs in the code, and can make it easier to maintain and update over time.\\n10. Testing of Microservices: Langsmith can be used to test individual microservices or entire microservices architectures. This can help ensure that each service is working as expected, and that the overall system is functioning correctly.'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#llm.invoke(\"how can langsmith help with testing?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5bcacc7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.prompts import ChatPromptTemplate\n",
    "prompt = ChatPromptTemplate.from_messages([\n",
    "    (\"system\", \"You are world class technical documentation writer.\"),\n",
    "    (\"user\", \"{input}\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f999550b",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = prompt | llm "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fdc8e4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#chain.invoke({\"input\":\"what does ollama pull <model_name> do?\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6a8edeb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.output_parsers import StrOutputParser\n",
    "\n",
    "output_parser = StrOutputParser()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "77d3258f",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = prompt | llm | output_parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2066f1d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "#chain.invoke({\"input\": \"how can langsmith help with testing?\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "edba5b7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import WebBaseLoader\n",
    "loader = WebBaseLoader(\"https://docs.smith.langchain.com/overview\")\n",
    "\n",
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9fcbbc3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.embeddings import OllamaEmbeddings\n",
    "\n",
    "embeddings = OllamaEmbeddings()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6a23106c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.vectorstores import DocArrayInMemorySearch\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter()\n",
    "documents = text_splitter.split_documents(docs)\n",
    "vector = DocArrayInMemorySearch.from_documents(documents, embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "62d5684f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains.combine_documents import create_stuff_documents_chain\n",
    "\n",
    "prompt = ChatPromptTemplate.from_template(\"\"\"Answer the following question based only on the provided context:\n",
    "\n",
    "<context>\n",
    "{context}\n",
    "</context>\n",
    "\n",
    "Question: {input}\"\"\")\n",
    "\n",
    "document_chain = create_stuff_documents_chain(llm, prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b8f9f0b7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nBased on the provided context, langsmith can help with testing by allowing users to visualize their test results. This means that users can use langsmith to display and analyze their test data in a graphical format, which can help them better understand the results and make more informed decisions.'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_core.documents import Document\n",
    "\n",
    "document_chain.invoke({\n",
    "    \"input\": \"how can langsmith help with testing?\",\n",
    "    \"context\": [Document(page_content=\"langsmith can let you visualize test results\")]\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "8dbf1db2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import create_retrieval_chain\n",
    "\n",
    "retriever = vector.as_retriever()\n",
    "retrieval_chain = create_retrieval_chain(retriever, document_chain)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "028a1851",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LangSmith can help with testing in several ways:\n",
      "\n",
      "1. Tracing: LangSmith provides detailed tracing of all LLM calls, which can help identify bottlenecks and issues in the chain or agent. By analyzing the traces, you can understand what is causing a slow response or unexpected output.\n",
      "2. Debugging: LangSmith's tracing feature helps debug LLM calls by providing visibility into the exact inputs and outputs of each call. This makes it easier to identify bugs and errors in the chain or agent.\n",
      "3. Testing: LangSmith provides a simple way to test LLMs, chains, and agents without requiring additional infrastructure. You can use LangSmith to generate test cases and evaluate the performance of your models under different conditions.\n",
      "4. Monitoring: LangSmith allows you to monitor the performance of your LLMs, chains, and agents in real-time. This helps identify issues early on and enables you to take corrective action before they impact your users.\n",
      "5. Collaborative debugging: LangSmith provides a collaborative debugging feature that allows you to share traces with your team for further analysis. This helps accelerate the debugging process and ensure that everyone is on the same page.\n",
      "6. Collecting examples: LangSmith allows you to collect examples of inputs and outputs, which can be used to test and evaluate your LLMs, chains, and agents. You can use these examples to train your models or to evaluate their performance under different conditions.\n",
      "7. Evaluating: LangSmith provides a simple way to evaluate the performance of your LLMs, chains, and agents using various metrics such as response time, accuracy, and F1 score. This helps you identify areas for improvement and optimize your models accordingly.\n",
      "8. Exporting datasets: LangSmith allows you to export traces and examples to popular formats such as CSV or JSON. This makes it easier to share data with your team or to use in other tools and platforms.\n",
      "\n",
      "By using LangSmith, you can significantly improve the testing and evaluation process for your LLMs, chains, and agents, which can ultimately lead to better performance and a better user experience.\n"
     ]
    }
   ],
   "source": [
    "response = retrieval_chain.invoke({\"input\": \"how can langsmith help with testing?\"})\n",
    "print(response[\"answer\"])\n",
    "\n",
    "# LangSmith offers several features that can help with testing:..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e2a8cc35",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import create_history_aware_retriever\n",
    "from langchain_core.prompts import MessagesPlaceholder\n",
    "\n",
    "# First we need a prompt that we can pass into an LLM to generate this search query\n",
    "\n",
    "prompt = ChatPromptTemplate.from_messages([\n",
    "    MessagesPlaceholder(variable_name=\"chat_history\"),\n",
    "    (\"user\", \"{input}\"),\n",
    "    (\"user\", \"Given the above conversation, generate a search query to look up in order to get information relevant to the conversation\")\n",
    "])\n",
    "retriever_chain = create_history_aware_retriever(llm, retriever, prompt) #retriever here is vectore store as a retriever"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "7b84ed30",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='inputs, and see what happens. At some point though, our application is performing\\nwell and we want to be more rigorous about testing changes. We can use a dataset\\nthat we’ve constructed along the way (see above). Alternatively, we could spend some\\ntime constructing a small dataset by hand. For these situations, LangSmith simplifies\\ndataset uploading.Once we have a dataset, how can we use it to test changes to a prompt or chain? The most basic approach is to run the chain over the data points and visualize the outputs. Despite technological advancements, there still is no substitute for looking at outputs by eye. Currently, running the chain over the data points needs to be done client-side. The LangSmith client makes it easy to pull down a dataset and then run a chain over them, logging the results to a new project associated with the dataset. From there, you can review them. We\\'ve made it easy to assign feedback to runs and mark them as correct or incorrect directly in the web app, displaying aggregate statistics for each test project.We also make it easier to evaluate these runs. To that end, we\\'ve added a set of evaluators to the open-source LangChain library. These evaluators can be specified when initiating a test run and will evaluate the results once the test run completes. If we’re being honest, most of these evaluators aren\\'t perfect. We would not recommend that you trust them blindly. However, we do think they are useful for guiding your eye to examples you should look at. This becomes especially valuable as the number of data points increases and it becomes infeasible to look at each one manually.Human Evaluation\\u200bAutomatic evaluation metrics are helpful for getting consistent and scalable information about model behavior;\\nhowever, there is still no complete substitute for human review to get the utmost quality and reliability from your application.\\nLangSmith makes it easy to manually review and annotate runs through annotation queues.These queues allow you to select any runs based on criteria like model type or automatic evaluation scores, and queue them up for human review. As a reviewer, you can then quickly step through the runs, viewing the input, output, and any existing tags before adding your own feedback.We often use this for a couple of reasons:To assess subjective qualities that automatic evaluators struggle with, like creativity or humorTo sample and validate a subset of runs that were auto-evaluated, ensuring the automatic metrics are still reliable and capturing the right informationAll the annotations made using the queue are assigned as \"feedback\" to the source runs, so you can easily filter and analyze them later.', metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'}),\n",
       " Document(page_content=\"Skip to main content🦜️🛠️ LangSmith DocsPython DocsJS/TS DocsSearchGo to AppLangSmithOverviewTracingTesting & EvaluationOrganizationsHubLangSmith CookbookOverviewOn this pageLangSmith Overview and User GuideBuilding reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.Over the past two months, we at LangChain have been building and using LangSmith with the goal of bridging this gap. This is our tactical user guide to outline effective ways to use LangSmith and maximize its benefits.On by default\\u200bAt LangChain, all of us have LangSmith’s tracing running in the background by default. On the Python side, this is achieved by setting environment variables, which we establish whenever we launch a virtual environment or open our bash shell and leave them set. The same principle applies to most JavaScript environments through process.env1.The benefit here is that all calls to LLMs, chains, agents, tools, and retrievers are logged to LangSmith. Around 90% of the time we don’t even look at the traces, but the 10% of the time that we do… it’s so helpful. We can use LangSmith to debug:An unexpected end resultWhy an agent is loopingWhy a chain was slower than expectedHow many tokens an agent usedDebugging\\u200bDebugging LLMs, chains, and agents can be tough. LangSmith helps solve the following pain points:What was the exact input to the LLM?\\u200bLLM calls are often tricky and non-deterministic. The inputs/outputs may seem straightforward, given they are technically string → string (or chat messages → chat message), but this can be misleading as the input string is usually constructed from a combination of user input and auxiliary functions.Most inputs to an LLM call are a combination of some type of fixed template along with input variables. These input variables could come directly from user input or from an auxiliary function (like retrieval). By the time these input variables go into the LLM they will have been converted to a string format, but often times they are not naturally represented as a string (they could be a list, or a Document object). Therefore, it is important to have visibility into what exactly the final string going into the LLM is. This has helped us debug bugs in formatting logic, unexpected transformations to user input, and straight up missing user input.To a much lesser extent, this is also true of the output of an LLM. Oftentimes the output of an LLM is technically a string but that string may contain some structure (json, yaml) that is intended to be parsed into a structured representation. Understanding what the exact output is can help determine if there may be a need for different parsing.LangSmith provides a straightforward visualization of the exact inputs/outputs to all LLM calls, so you can easily understand them.If I edit the prompt, how does that affect the output?\\u200bSo you notice a bad output, and you go into LangSmith to see what's going on. You find the faulty LLM call and are now looking at the exact input. You want to try changing a word or a phrase to see what happens -- what do you do?We constantly ran into this issue. Initially, we copied the prompt to a playground of sorts. But this got annoying, so we built a playground of our own! When examining an LLM call, you can click the Open in Playground button to access this playground. Here, you can modify the prompt and re-run it to observe the resulting changes to the output - as many times as needed!Currently, this feature supports only OpenAI and Anthropic models and works for LLM and Chat Model calls. We plan to extend its functionality to more LLM types, chains, agents, and retrievers in the future.What is the exact sequence of events?\\u200bIn complicated chains and agents, it can often be hard to understand what is going on under the hood. What calls are being made? In what\", metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'}),\n",
       " Document(page_content=\"retrievers in the future.What is the exact sequence of events?\\u200bIn complicated chains and agents, it can often be hard to understand what is going on under the hood. What calls are being made? In what order? What are the inputs and outputs of each call?LangSmith's built-in tracing feature offers a visualization to clarify these sequences. This tool is invaluable for understanding intricate and lengthy chains and agents. For chains, it can shed light on the sequence of calls and how they interact. For agents, where the sequence of calls is non-deterministic, it helps visualize the specific sequence for a given run -- something that is impossible to know ahead of time.Why did my chain take much longer than expected?\\u200bIf a chain takes longer than expected, you need to identify the cause. By tracking the latency of each step, LangSmith lets you identify and possibly eliminate the slowest components.How many tokens were used?\\u200bBuilding and prototyping LLM applications can be expensive. LangSmith tracks the total token usage for a chain and the token usage of each step. This makes it easy to identify potentially costly parts of the chain.Collaborative debugging\\u200bIn the past, sharing a faulty chain with a colleague for debugging was challenging when performed locally. With LangSmith, we've added a “Share” button that makes the chain and LLM runs accessible to anyone with the shared link.Collecting examples\\u200bMost of the time we go to debug, it's because something bad or unexpected outcome has happened in our application. These failures are valuable data points! By identifying how our chain can fail and monitoring these failures, we can test future chain versions against these known issues.Why is this so impactful? When building LLM applications, it’s often common to start without a dataset of any kind. This is part of the power of LLMs! They are amazing zero-shot learners, making it possible to get started as easily as possible. But this can also be a curse -- as you adjust the prompt, you're wandering blind. You don’t have any examples to benchmark your changes against.LangSmith addresses this problem by including an “Add to Dataset” button for each run, making it easy to add the input/output examples a chosen dataset. You can edit the example before adding them to the dataset to include the expected result, which is particularly useful for bad examples.This feature is available at every step of a nested chain, enabling you to add examples for an end-to-end chain, an intermediary chain (such as a LLM Chain), or simply the LLM or Chat Model.End-to-end chain examples are excellent for testing the overall flow, while single, modular LLM Chain or LLM/Chat Model examples can be beneficial for testing the simplest and most directly modifiable components.Testing & evaluation\\u200bInitially, we do most of our evaluation manually and ad hoc. We pass in different\", metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'}),\n",
       " Document(page_content=\"You can also quickly edit examples and add them to datasets to expand the surface area of your evaluation sets or to fine-tune a model for improved quality or reduced costs.Monitoring\\u200bAfter all this, your app might finally ready to go in production. LangSmith can also be used to monitor your application in much the same way that you used for debugging. You can log all traces, visualize latency and token usage statistics, and troubleshoot specific issues as they arise. Each run can also be assigned string tags or key-value metadata, allowing you to attach correlation ids or AB test variants, and filter runs accordingly.We’ve also made it possible to associate feedback programmatically with runs. This means that if your application has a thumbs up/down button on it, you can use that to log feedback back to LangSmith. This can be used to track performance over time and pinpoint under performing data points, which you can subsequently add to a dataset for future testing — mirroring the debug mode approach.We’ve provided several examples in the LangSmith documentation for extracting insights from logged runs. In addition to guiding you on performing this task yourself, we also provide examples of integrating with third parties for this purpose. We're eager to expand this area in the coming months! If you have ideas for either -- an open-source way to evaluate, or are building a company that wants to do analytics over these runs, please reach out.Exporting datasets\\u200bLangSmith makes it easy to curate datasets. However, these aren’t just useful inside LangSmith; they can be exported for use in other contexts. Notable applications include exporting for use in OpenAI Evals or fine-tuning, such as with FireworksAI.To set up tracing in Deno, web browsers, or other runtime environments without access to the environment, check out the FAQs.↩PreviousLangSmithNextTracingOn by defaultDebuggingWhat was the exact input to the LLM?If I edit the prompt, how does that affect the output?What is the exact sequence of events?Why did my chain take much longer than expected?How many tokens were used?Collaborative debuggingCollecting examplesTesting & evaluationHuman EvaluationMonitoringExporting datasetsCommunityDiscordTwitterGitHubDocs CodeLangSmith SDKPythonJS/TSMoreHomepageBlogCopyright © 2023 LangChain, Inc.\", metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'})]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_core.messages import HumanMessage, AIMessage\n",
    "\n",
    "chat_history = [HumanMessage(content=\"Can LangSmith help test my LLM applications?\"), AIMessage(content=\"Yes!\")]\n",
    "retriever_chain.invoke({\n",
    "    \"chat_history\": chat_history,\n",
    "    \"input\": \"Tell me how\"\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7d20db9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:langchain]",
   "language": "python",
   "name": "conda-env-langchain-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}