Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Implement parallel reading of Parquet footers without using the additional thread pool #18082

Open
vuule opened this issue Feb 24, 2025 · 0 comments
Assignees
Labels
feature request New feature or request

Comments

@vuule
Copy link
Contributor

vuule commented Feb 24, 2025

#17957 optimizes ingest of multiple Parquet files, using an internal thread pool to issue read requests. Parallelism of the read operations is already implemented in kvikIO, so we should not need to launch these reads from multiple threads.

With an async read interface, we could read all footers in a loop from a single thread.

@vuule vuule added the feature request New feature or request label Feb 24, 2025
@vuule vuule self-assigned this Feb 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant