Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

File IO on Tokio Workers #8188

Open
tustvold opened this issue Nov 15, 2023 · 0 comments
Open

File IO on Tokio Workers #8188

tustvold opened this issue Nov 15, 2023 · 0 comments
Assignees
@tustvold
Labels
bug Something isn't working

Comments

@tustvold
Copy link
Contributor

tustvold commented Nov 15, 2023 •
edited
Loading

Describe the bug

Both CsvExec and NdJsonExec perform file IO on the tokio worker threads, this acts to starve the worker pool, potentially leaving CPUs idle. In cases involving FIFO files, this could even lead to deadlock

To Reproduce

No response

Expected behavior

We should probably just convert the files to streams and decode them that way

Additional context

We probably want to wait for #7994 first, so that any changes don't negatively impact the streaming use-case
@tustvold tustvold added the bug Something isn't working label Nov 15, 2023
@tustvold tustvold self-assigned this Nov 15, 2023
@tustvold tustvold mentioned this issue Nov 15, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tustvold tustvold

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tustvold