-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs[minor]: updated DirectoryLoader docs (#6347)
* docs[minor]: updated DirectoryLoader docs * cr
- Loading branch information
1 parent
668588e
commit 866e8e0
Showing
2 changed files
with
192 additions
and
42 deletions.
There are no files selected for viewing
192 changes: 192 additions & 0 deletions
192
docs/core_docs/docs/integrations/document_loaders/file_loaders/directory.ipynb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,192 @@ | ||
{ | ||
"cells": [ | ||
{ | ||
"cell_type": "raw", | ||
"metadata": {}, | ||
"source": [ | ||
"---\n", | ||
"sidebar_label: DirectoryLoader\n", | ||
"sidebar_class_name: node-only\n", | ||
"---" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"# DirectoryLoader\n", | ||
"\n", | ||
"```{=mdx}\n", | ||
"\n", | ||
":::tip Compatibility\n", | ||
"\n", | ||
"Only available on Node.js.\n", | ||
"\n", | ||
":::\n", | ||
"\n", | ||
"```\n", | ||
"\n", | ||
"This notebook provides a quick overview for getting started with `DirectoryLoader` [document loaders](/docs/concepts/#document-loaders). For detailed documentation of all `DirectoryLoader` features and configurations head to the [API reference](https://api.js.langchain.com/classes/langchain_document_loaders_fs_directory.DirectoryLoader.html).\n", | ||
"\n", | ||
"This example goes over how to load data from folders with multiple files. The second argument is a map of file extensions to loader factories. Each file will be passed to the matching loader, and the resulting documents will be concatenated together.\n", | ||
"\n", | ||
"Example folder:\n", | ||
"\n", | ||
"```text\n", | ||
"src/document_loaders/example_data/example/\n", | ||
"├── example.json\n", | ||
"├── example.jsonl\n", | ||
"├── example.txt\n", | ||
"└── example.csv\n", | ||
"```\n", | ||
"\n", | ||
"## Overview\n", | ||
"### Integration details\n", | ||
"\n", | ||
"| Class | Package | Compatibility | Local | PY support | \n", | ||
"| :--- | :--- | :---: | :---: | :---: |\n", | ||
"| [DirectoryLoader](https://api.js.langchain.com/classes/langchain_document_loaders_fs_directory.DirectoryLoader.html) | [langchain](https://api.js.langchain.com/modules/langchain_document_loaders_fs_directory.html) | Node-only | ✅ | ✅ |\n", | ||
"\n", | ||
"## Setup\n", | ||
"\n", | ||
"To access `DirectoryLoader` document loader you'll need to install the `langchain` package.\n", | ||
"\n", | ||
"### Installation\n", | ||
"\n", | ||
"The LangChain DirectoryLoader integration lives in the `langchain` package:\n", | ||
"\n", | ||
"```{=mdx}\n", | ||
"import IntegrationInstallTooltip from \"@mdx_components/integration_install_tooltip.mdx\";\n", | ||
"import Npm2Yarn from \"@theme/Npm2Yarn\";\n", | ||
"\n", | ||
"<IntegrationInstallTooltip></IntegrationInstallTooltip>\n", | ||
"\n", | ||
"<Npm2Yarn>\n", | ||
" langchain\n", | ||
"</Npm2Yarn>\n", | ||
"\n", | ||
"```" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"## Instantiation\n", | ||
"\n", | ||
"Now we can instantiate our model object and load documents:" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 1, | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"import { DirectoryLoader } from \"langchain/document_loaders/fs/directory\";\n", | ||
"import {\n", | ||
" JSONLoader,\n", | ||
" JSONLinesLoader,\n", | ||
"} from \"langchain/document_loaders/fs/json\";\n", | ||
"import { TextLoader } from \"langchain/document_loaders/fs/text\";\n", | ||
"import { CSVLoader } from \"@langchain/community/document_loaders/fs/csv\";\n", | ||
"\n", | ||
"const loader = new DirectoryLoader(\n", | ||
" \"../../../../../../examples/src/document_loaders/example_data\",\n", | ||
" {\n", | ||
" \".json\": (path) => new JSONLoader(path, \"/texts\"),\n", | ||
" \".jsonl\": (path) => new JSONLinesLoader(path, \"/html\"),\n", | ||
" \".txt\": (path) => new TextLoader(path),\n", | ||
" \".csv\": (path) => new CSVLoader(path, \"text\"),\n", | ||
" }\n", | ||
");" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"## Load" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 2, | ||
"metadata": {}, | ||
"outputs": [ | ||
{ | ||
"name": "stdout", | ||
"output_type": "stream", | ||
"text": [ | ||
"Document {\n", | ||
" pageContent: 'Foo\\nBar\\nBaz\\n\\n',\n", | ||
" metadata: {\n", | ||
" source: '/Users/bracesproul/code/lang-chain-ai/langchainjs/examples/src/document_loaders/example_data/example.txt'\n", | ||
" },\n", | ||
" id: undefined\n", | ||
"}\n" | ||
] | ||
} | ||
], | ||
"source": [ | ||
"const docs = await loader.load()\n", | ||
"// disable console.warn calls\n", | ||
"console.warn = () => {}\n", | ||
"docs[0]" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": 7, | ||
"metadata": {}, | ||
"outputs": [ | ||
{ | ||
"name": "stdout", | ||
"output_type": "stream", | ||
"text": [ | ||
"{\n", | ||
" source: '/Users/bracesproul/code/lang-chain-ai/langchainjs/examples/src/document_loaders/example_data/example.txt'\n", | ||
"}\n" | ||
] | ||
} | ||
], | ||
"source": [ | ||
"console.log(docs[0].metadata)" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"## API reference\n", | ||
"\n", | ||
"For detailed documentation of all DirectoryLoader features and configurations head to the API reference: https://api.js.langchain.com/classes/langchain_document_loaders_fs_directory.DirectoryLoader.html" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [] | ||
} | ||
], | ||
"metadata": { | ||
"kernelspec": { | ||
"display_name": "TypeScript", | ||
"language": "typescript", | ||
"name": "tslab" | ||
}, | ||
"language_info": { | ||
"codemirror_mode": { | ||
"mode": "typescript", | ||
"name": "javascript", | ||
"typescript": true | ||
}, | ||
"file_extension": ".ts", | ||
"mimetype": "text/typescript", | ||
"name": "typescript", | ||
"version": "3.7.2" | ||
} | ||
}, | ||
"nbformat": 4, | ||
"nbformat_minor": 4 | ||
} |
42 changes: 0 additions & 42 deletions
42
docs/core_docs/docs/integrations/document_loaders/file_loaders/directory.mdx
This file was deleted.
Oops, something went wrong.