-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs[patch]: Adds docs on cancelling execution (#6364)
* Adds docs on cancelling execution * Fix build, remove stray TODOs * Fix regex * Fix * Update regex * Test * log * fix * Fix
- Loading branch information
1 parent
dbcf167
commit fada375
Showing
6 changed files
with
347 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,297 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# How to cancel execution\n", | ||
| "\n", | ||
| "```{=mdx}\n", | ||
| ":::info Prerequisites\n", | ||
| "\n", | ||
| "This guide assumes familiarity with the following concepts:\n", | ||
| "\n", | ||
| "- [LangChain Expression Language](/docs/concepts/#langchain-expression-language)\n", | ||
| "- [Chains](/docs/how_to/sequence/)\n", | ||
| "- [Streaming](/docs/how_to/streaming/)\n", | ||
| "\n", | ||
| ":::\n", | ||
| "```\n", | ||
| "\n", | ||
| "When building longer-running chains or [LangGraph](https://langchain-ai.github.io/langgraphjs/) agents, you may want to interrupt execution in situations such as a user leaving your app or submitting a new query.\n", | ||
| "\n", | ||
| "[LangChain Expression Language (LCEL)](/docs/concepts#langchain-expression-language) supports aborting runnables that are in-progress via a runtime [signal](https://developer.mozilla.org/en-US/docs/Web/API/AbortController/signal) option.\n", | ||
| "\n", | ||
| "```{=mdx}\n", | ||
| ":::caution Compatibility\n", | ||
| "\n", | ||
| "Built-in signal support requires `@langchain/core>=0.2.20`. Please see here for a [guide on upgrading](/docs/how_to/installation/#installing-integration-packages).\n", | ||
| "\n", | ||
| ":::\n", | ||
| "```\n", | ||
| "\n", | ||
| "**Note:** Individual integrations like chat models or retrievers may have missing or differing implementations for aborting execution. Signal support as described in this guide will apply in between steps of a chain.\n", | ||
| "\n", | ||
| "To see how this works, construct a chain such as the one below that performs [retrieval-augmented generation](/docs/tutorials/rag). It answers questions by first searching the web using [Tavily](/docs/integrations/retrievers/tavily), then passing the results to a chat model to generate a final answer:\n", | ||
| "\n", | ||
| "```{=mdx}\n", | ||
| "import ChatModelTabs from \"@theme/ChatModelTabs\";\n", | ||
| "\n", | ||
| "<ChatModelTabs />\n", | ||
| "```" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 1, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "// @lc-docs-hide-cell\n", | ||
| "import { ChatAnthropic } from \"@langchain/anthropic\";\n", | ||
| "\n", | ||
| "const llm = new ChatAnthropic({\n", | ||
| " model: \"claude-3-5-sonnet-20240620\",\n", | ||
| "});" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 2, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import { TavilySearchAPIRetriever } from \"@langchain/community/retrievers/tavily_search_api\";\n", | ||
| "import type { Document } from \"@langchain/core/documents\";\n", | ||
| "import { StringOutputParser } from \"@langchain/core/output_parsers\";\n", | ||
| "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n", | ||
| "import { RunnablePassthrough, RunnableSequence } from \"@langchain/core/runnables\";\n", | ||
| "\n", | ||
| "const formatDocsAsString = (docs: Document[]) => {\n", | ||
| " return docs.map((doc) => doc.pageContent).join(\"\\n\\n\")\n", | ||
| "}\n", | ||
| "\n", | ||
| "const retriever = new TavilySearchAPIRetriever({\n", | ||
| " k: 3,\n", | ||
| "});\n", | ||
| "\n", | ||
| "const prompt = ChatPromptTemplate.fromTemplate(`\n", | ||
| "Use the following context to answer questions to the best of your ability:\n", | ||
| "\n", | ||
| "<context>\n", | ||
| "{context}\n", | ||
| "</context>\n", | ||
| "\n", | ||
| "Question: {question}`)\n", | ||
| "\n", | ||
| "const chain = RunnableSequence.from([\n", | ||
| " {\n", | ||
| " context: retriever.pipe(formatDocsAsString),\n", | ||
| " question: new RunnablePassthrough(),\n", | ||
| " },\n", | ||
| " prompt,\n", | ||
| " llm,\n", | ||
| " new StringOutputParser(),\n", | ||
| "]);" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "If you invoke it normally, you can see it returns up-to-date information:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 3, | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "Based on the provided context, the current weather in San Francisco is:\n", | ||
| "\n", | ||
| "Temperature: 17.6°C (63.7°F)\n", | ||
| "Condition: Sunny\n", | ||
| "Wind: 14.4 km/h (8.9 mph) from WSW direction\n", | ||
| "Humidity: 74%\n", | ||
| "Cloud cover: 15%\n", | ||
| "\n", | ||
| "The information indicates it's a sunny day with mild temperatures and light winds. The data appears to be from August 2, 2024, at 17:00 local time.\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "await chain.invoke(\"what is the current weather in SF?\");" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Now, let's interrupt it early. Initialize an [`AbortController`](https://developer.mozilla.org/en-US/docs/Web/API/AbortController) and pass its `signal` property into the chain execution. To illustrate the fact that the cancellation occurs as soon as possible, set a timeout of 100ms:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 4, | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "Error: Aborted\n", | ||
| " at EventTarget.<anonymous> (/Users/jacoblee/langchain/langchainjs/langchain-core/dist/utils/signal.cjs:19:24)\n", | ||
| " at [nodejs.internal.kHybridDispatch] (node:internal/event_target:825:20)\n", | ||
| " at EventTarget.dispatchEvent (node:internal/event_target:760:26)\n", | ||
| " at abortSignal (node:internal/abort_controller:370:10)\n", | ||
| " at AbortController.abort (node:internal/abort_controller:392:5)\n", | ||
| " at Timeout._onTimeout (evalmachine.<anonymous>:7:29)\n", | ||
| " at listOnTimeout (node:internal/timers:573:17)\n", | ||
| " at process.processTimers (node:internal/timers:514:7)\n", | ||
| "timer1: 103.204ms\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "const controller = new AbortController();\n", | ||
| "\n", | ||
| "const startTimer = console.time(\"timer1\");\n", | ||
| "\n", | ||
| "setTimeout(() => controller.abort(), 100);\n", | ||
| "\n", | ||
| "try {\n", | ||
| " await chain.invoke(\"what is the current weather in SF?\", {\n", | ||
| " signal: controller.signal,\n", | ||
| " });\n", | ||
| "} catch (e) {\n", | ||
| " console.log(e);\n", | ||
| "}\n", | ||
| "\n", | ||
| "console.timeEnd(\"timer1\");" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "And you can see that execution ends after just over 100ms. Looking at [this LangSmith trace](https://smith.langchain.com/public/63c04c3b-2683-4b73-a4f7-fb12f5cb9180/r), you can see that the model is never called.\n", | ||
| "\n", | ||
| "## Streaming\n", | ||
| "\n", | ||
| "You can pass a `signal` when streaming too. This gives you more control over using a `break` statement within the `for await... of` loop to cancel the current run, which will only trigger after final output has already started streaming. The below example uses a `break` statement - note the time elapsed before cancellation occurs:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 5, | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "chunk \n", | ||
| "timer2: 3.990s\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "const startTimer2 = console.time(\"timer2\");\n", | ||
| "\n", | ||
| "const stream = await chain.stream(\"what is the current weather in SF?\");\n", | ||
| "\n", | ||
| "for await (const chunk of stream) {\n", | ||
| " console.log(\"chunk\", chunk);\n", | ||
| " break;\n", | ||
| "}\n", | ||
| "\n", | ||
| "console.timeEnd(\"timer2\");" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Now compare this to using a signal. Note that you will need to wrap the stream in a `try/catch` block:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 6, | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "Error: Aborted\n", | ||
| " at EventTarget.<anonymous> (/Users/jacoblee/langchain/langchainjs/langchain-core/dist/utils/signal.cjs:19:24)\n", | ||
| " at [nodejs.internal.kHybridDispatch] (node:internal/event_target:825:20)\n", | ||
| " at EventTarget.dispatchEvent (node:internal/event_target:760:26)\n", | ||
| " at abortSignal (node:internal/abort_controller:370:10)\n", | ||
| " at AbortController.abort (node:internal/abort_controller:392:5)\n", | ||
| " at Timeout._onTimeout (evalmachine.<anonymous>:7:38)\n", | ||
| " at listOnTimeout (node:internal/timers:573:17)\n", | ||
| " at process.processTimers (node:internal/timers:514:7)\n", | ||
| "timer3: 100.684ms\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "const controllerForStream = new AbortController();\n", | ||
| "\n", | ||
| "const startTimer3 = console.time(\"timer3\");\n", | ||
| "\n", | ||
| "setTimeout(() => controllerForStream.abort(), 100);\n", | ||
| "\n", | ||
| "try {\n", | ||
| " const streamWithSignal = await chain.stream(\"what is the current weather in SF?\", {\n", | ||
| " signal: controllerForStream.signal\n", | ||
| " });\n", | ||
| " for await (const chunk of streamWithSignal) {\n", | ||
| " console.log(chunk);\n", | ||
| " break;\n", | ||
| " } \n", | ||
| "} catch (e) {\n", | ||
| " console.log(e); \n", | ||
| "}\n", | ||
| "\n", | ||
| "console.timeEnd(\"timer3\");" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Related\n", | ||
| "\n", | ||
| "- [Pass through arguments from one step to the next](/docs/how_to/passthrough)\n", | ||
| "- [Dispatching custom events](/docs/how_to/callbacks_custom_events)" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "TypeScript", | ||
| "language": "typescript", | ||
| "name": "tslab" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "mode": "typescript", | ||
| "name": "javascript", | ||
| "typescript": true | ||
| }, | ||
| "file_extension": ".ts", | ||
| "mimetype": "text/typescript", | ||
| "name": "typescript", | ||
| "version": "3.7.2" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 2 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.