Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

community: implement the atransform_documents method for MarkdownifyTransformer #27866

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

rparkr
Copy link

@rparkr rparkr commented Nov 3, 2024

Description

Implements the atransform_documents method for MarkdownifyTransformer using the asyncio built-in library for concurrency.

Issue

Fixes #27865

Dependencies

No new dependencies added, but markdownify is required since this PR updates the markdownify integration.

Tests and docs

  • Tests not yet added but can be done soon
  • I did not modify the docstrings since they already described the basic functionality, and the API docs also already included a description. If it would be helpful, I would be happy to update the docstrings and/or the API docs.

Lint and test

  • format
  • lint
  • test

I have not yet run formatting, linting, and tests, but I can do that soon.

- Add asynchronous method for transforming a single document (`_atransform_document`)
- Implement the asynchronous method for transforming a list of documents using async.gather
- Remove extra function definition; replace with asyncio.create_task()
@dosubot dosubot bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Nov 3, 2024
Copy link

vercel bot commented Nov 3, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
langchain ⬜️ Ignored (Inspect) Visit Preview Nov 3, 2024 2:33am

@dosubot dosubot bot added the community Related to langchain-community label Nov 3, 2024
- Split the list comprehension across lines for better readability
@rparkr rparkr marked this pull request as draft November 3, 2024 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community Related to langchain-community size:M This PR changes 30-99 lines, ignoring generated files.
Projects
Status: Triage
1 participant