langchain[patch],community[patch]: add in-memory option for unstructured loader #5581
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
dosubot
bot
added
the
size:M
dosubot
bot
added
size:L
![dosubot[bot]](https://avatars.githubusercontent.com/in/324583?s=60&v=4){kind=small}
| @@ -29,6 +30,34 @@ test.skip("Test Unstructured base loader", async () => { | |||
| } | |||
There was a problem hiding this comment.
Hey team, just a heads up that I've flagged a change in the PR related to accessing an environment variable using process.env. This is for your review to ensure proper handling of environment variables. Keep up the great work!
There was a problem hiding this comment.
that's right, I've run the tests locally using a docker instance of unstructured
| @@ -175,19 +184,30 @@ export class UnstructuredLoader extends BaseDocumentLoader { | |||
| private maxCharacters?: number; | |||
|
|
|||
| constructor( | |||
| filePathOrLegacyApiUrl: string, | |||
| filePathOrLegacyApiUrlOrMemoryBuffer: | |||
There was a problem hiding this comment.
We reeealy need to clean this up 😬
|
Looks reasonable, thank you! |
|
Can't push to your branch - this lint rule is pretty silly, so you can either fix it or just add |
|
@jacoblee93 fixed the formatting |
|
Thank you! |
|
Merged in #5726 |
Fixes #1618
Add support for passing buffer to unstructured loader.
This covers cases when fetching a file externally using serverless functions. Without this change we always have to write a new file before passing the path to the loader. This is an anti pattern for serverless environments, such as Vercel.
https://github.com/orgs/vercel/discussions/241#discussioncomment-363002
Old implementation was already loading the file in memory.
Fixes # (issue)
twitter @andrewdorobantu