Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Fixed limit pushdown in parquet #1180

Merged
merged 2 commits into from
Jul 25, 2022
Merged

Fixed limit pushdown in parquet #1180

merged 2 commits into from
Jul 25, 2022

Conversation

jorgecarleitao
Copy link
Owner

The limit was not correctly applied within a row group, only across row groups, resulting in un-necessary CPU work and memory usage.

@jorgecarleitao jorgecarleitao added the bug Something isn't working label Jul 24, 2022
@codecov
Copy link

codecov bot commented Jul 24, 2022

Codecov Report

Merging #1180 (aac776c) into main (f9562e0) will increase coverage by 0.08%.
The diff coverage is 87.61%.

@@            Coverage Diff             @@
##             main    #1180      +/-   ##
==========================================
+ Coverage   83.38%   83.47%   +0.08%     
==========================================
  Files         363      363              
  Lines       36272    36490     +218     
==========================================
+ Hits        30247    30461     +214     
- Misses       6025     6029       +4     
Impacted Files Coverage Δ
...t/read/deserialize/fixed_size_binary/dictionary.rs 40.20% <37.50%> (+0.64%) ⬆️
src/io/parquet/read/deserialize/nested_utils.rs 83.43% <52.17%> (-0.81%) ⬇️
src/io/parquet/read/deserialize/mod.rs 72.02% <69.38%> (-0.26%) ⬇️
src/io/parquet/read/deserialize/utils.rs 78.87% <89.47%> (+0.40%) ⬆️
src/io/parquet/read/deserialize/simple.rs 83.88% <90.90%> (+1.04%) ⬆️
src/io/parquet/read/deserialize/binary/basic.rs 75.39% <100.00%> (-0.20%) ⬇️
...c/io/parquet/read/deserialize/binary/dictionary.rs 92.79% <100.00%> (+2.31%) ⬆️
src/io/parquet/read/deserialize/binary/nested.rs 80.14% <100.00%> (+0.73%) ⬆️
src/io/parquet/read/deserialize/boolean/basic.rs 93.49% <100.00%> (-0.72%) ⬇️
src/io/parquet/read/deserialize/boolean/nested.rs 77.08% <100.00%> (+1.25%) ⬆️
... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f9562e0...aac776c. Read the comment docs.

@jorgecarleitao jorgecarleitao merged commit e810d39 into main Jul 25, 2022
@jorgecarleitao jorgecarleitao deleted the fix_limit branch July 25, 2022 04:50
@eddyxu eddyxu mentioned this pull request Aug 1, 2022
5 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant