Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Added support for limit pushdown to IPC reading #1135

Merged
merged 3 commits into from
Jul 5, 2022
Merged

Conversation

jorgecarleitao
Copy link
Owner

This PR adds Option<usize> argument to FileReader and FileStream to restrict the maximum number of read rows.

This is a typical optimization in query engines, but they do require the reader to support it.

@jorgecarleitao jorgecarleitao added the feature A new feature label Jul 1, 2022
@codecov
Copy link

codecov bot commented Jul 1, 2022

Codecov Report

Merging #1135 (1e34a84) into main (b3583b6) will increase coverage by 0.02%.
The diff coverage is 96.84%.

@@            Coverage Diff             @@
##             main    #1135      +/-   ##
==========================================
+ Coverage   83.49%   83.51%   +0.02%     
==========================================
  Files         366      366              
  Lines       35635    35721      +86     
==========================================
+ Hits        29752    29831      +79     
- Misses       5883     5890       +7     
Impacted Files Coverage Δ
src/io/ipc/mod.rs 100.00% <ø> (ø)
src/io/ipc/write/file_async.rs 81.75% <ø> (ø)
src/io/ipc/read/file_async.rs 61.19% <76.92%> (+0.49%) ⬆️
src/io/flight/mod.rs 67.32% <100.00%> (+0.32%) ⬆️
src/io/ipc/read/array/binary.rs 92.75% <100.00%> (+0.32%) ⬆️
src/io/ipc/read/array/boolean.rs 90.74% <100.00%> (+0.54%) ⬆️
src/io/ipc/read/array/dictionary.rs 87.23% <100.00%> (+0.56%) ⬆️
src/io/ipc/read/array/fixed_size_binary.rs 87.71% <100.00%> (+0.68%) ⬆️
src/io/ipc/read/array/fixed_size_list.rs 88.52% <100.00%> (+1.02%) ⬆️
src/io/ipc/read/array/list.rs 79.01% <100.00%> (+1.67%) ⬆️
... and 17 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b3583b6...1e34a84. Read the comment docs.

@jorgecarleitao jorgecarleitao merged commit 55e5924 into main Jul 5, 2022
@jorgecarleitao jorgecarleitao deleted the limit_pushdown branch July 5, 2022 14:39
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature A new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant