Skip to content

Commit

Permalink
docs[minor]: updated DirectoryLoader docs (#6347)
Browse files Browse the repository at this point in the history
* docs[minor]: updated DirectoryLoader docs

* cr
  • Loading branch information
bracesproul authored Aug 2, 2024
1 parent 668588e commit 866e8e0
Show file tree
Hide file tree
Showing 2 changed files with 192 additions and 42 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,192 @@
{
"cells": [
{
"cell_type": "raw",
"metadata": {},
"source": [
"---\n",
"sidebar_label: DirectoryLoader\n",
"sidebar_class_name: node-only\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# DirectoryLoader\n",
"\n",
"```{=mdx}\n",
"\n",
":::tip Compatibility\n",
"\n",
"Only available on Node.js.\n",
"\n",
":::\n",
"\n",
"```\n",
"\n",
"This notebook provides a quick overview for getting started with `DirectoryLoader` [document loaders](/docs/concepts/#document-loaders). For detailed documentation of all `DirectoryLoader` features and configurations head to the [API reference](https://api.js.langchain.com/classes/langchain_document_loaders_fs_directory.DirectoryLoader.html).\n",
"\n",
"This example goes over how to load data from folders with multiple files. The second argument is a map of file extensions to loader factories. Each file will be passed to the matching loader, and the resulting documents will be concatenated together.\n",
"\n",
"Example folder:\n",
"\n",
"```text\n",
"src/document_loaders/example_data/example/\n",
"├── example.json\n",
"├── example.jsonl\n",
"├── example.txt\n",
"└── example.csv\n",
"```\n",
"\n",
"## Overview\n",
"### Integration details\n",
"\n",
"| Class | Package | Compatibility | Local | PY support | \n",
"| :--- | :--- | :---: | :---: | :---: |\n",
"| [DirectoryLoader](https://api.js.langchain.com/classes/langchain_document_loaders_fs_directory.DirectoryLoader.html) | [langchain](https://api.js.langchain.com/modules/langchain_document_loaders_fs_directory.html) | Node-only | ✅ | ✅ |\n",
"\n",
"## Setup\n",
"\n",
"To access `DirectoryLoader` document loader you'll need to install the `langchain` package.\n",
"\n",
"### Installation\n",
"\n",
"The LangChain DirectoryLoader integration lives in the `langchain` package:\n",
"\n",
"```{=mdx}\n",
"import IntegrationInstallTooltip from \"@mdx_components/integration_install_tooltip.mdx\";\n",
"import Npm2Yarn from \"@theme/Npm2Yarn\";\n",
"\n",
"<IntegrationInstallTooltip></IntegrationInstallTooltip>\n",
"\n",
"<Npm2Yarn>\n",
" langchain\n",
"</Npm2Yarn>\n",
"\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Instantiation\n",
"\n",
"Now we can instantiate our model object and load documents:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import { DirectoryLoader } from \"langchain/document_loaders/fs/directory\";\n",
"import {\n",
" JSONLoader,\n",
" JSONLinesLoader,\n",
"} from \"langchain/document_loaders/fs/json\";\n",
"import { TextLoader } from \"langchain/document_loaders/fs/text\";\n",
"import { CSVLoader } from \"@langchain/community/document_loaders/fs/csv\";\n",
"\n",
"const loader = new DirectoryLoader(\n",
" \"../../../../../../examples/src/document_loaders/example_data\",\n",
" {\n",
" \".json\": (path) => new JSONLoader(path, \"/texts\"),\n",
" \".jsonl\": (path) => new JSONLinesLoader(path, \"/html\"),\n",
" \".txt\": (path) => new TextLoader(path),\n",
" \".csv\": (path) => new CSVLoader(path, \"text\"),\n",
" }\n",
");"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Document {\n",
" pageContent: 'Foo\\nBar\\nBaz\\n\\n',\n",
" metadata: {\n",
" source: '/Users/bracesproul/code/lang-chain-ai/langchainjs/examples/src/document_loaders/example_data/example.txt'\n",
" },\n",
" id: undefined\n",
"}\n"
]
}
],
"source": [
"const docs = await loader.load()\n",
"// disable console.warn calls\n",
"console.warn = () => {}\n",
"docs[0]"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" source: '/Users/bracesproul/code/lang-chain-ai/langchainjs/examples/src/document_loaders/example_data/example.txt'\n",
"}\n"
]
}
],
"source": [
"console.log(docs[0].metadata)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## API reference\n",
"\n",
"For detailed documentation of all DirectoryLoader features and configurations head to the API reference: https://api.js.langchain.com/classes/langchain_document_loaders_fs_directory.DirectoryLoader.html"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "TypeScript",
"language": "typescript",
"name": "tslab"
},
"language_info": {
"codemirror_mode": {
"mode": "typescript",
"name": "javascript",
"typescript": true
},
"file_extension": ".ts",
"mimetype": "text/typescript",
"name": "typescript",
"version": "3.7.2"
}
},
"nbformat": 4,
"nbformat_minor": 4
}

This file was deleted.

0 comments on commit 866e8e0

Please sign in to comment.