Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update arrow: ipc limit and reduce categorical-> dictionary bound checks #3926

Merged
merged 1 commit into from
Jul 8, 2022

Conversation

ritchie46
Copy link
Member

This adds upstream limit pushdown in the reader. Lazy scans that push the limit to the reader will now have faster reads and reduced memory.

This also removes some unneeded bound checks in converting from categorical to arrow dictionaries.

@github-actions github-actions bot added python Related to Python Polars rust Related to Rust Polars labels Jul 7, 2022
@codecov-commenter
Copy link

codecov-commenter commented Jul 7, 2022

Codecov Report

Merging #3926 (5b0e5e2) into master (d248d00) will increase coverage by 15.62%.
The diff coverage is 42.55%.

@@             Coverage Diff             @@
##           master    #3926       +/-   ##
===========================================
+ Coverage   62.42%   78.04%   +15.62%     
===========================================
  Files         447      448        +1     
  Lines       73817    74144      +327     
===========================================
+ Hits        46079    57866    +11787     
+ Misses      27738    16278    -11460     
Impacted Files Coverage Δ
polars/polars-io/src/parquet/write.rs 99.01% <ø> (ø)
py-polars/polars/internals/frame.py 93.19% <ø> (ø)
...core/src/chunked_array/logical/categorical/from.rs 27.41% <27.02%> (+5.79%) ⬆️
polars/polars-io/src/ipc/ipc_file.rs 94.42% <100.00%> (+45.84%) ⬆️
...olars-lazy/src/physical_plan/expressions/column.rs 63.63% <0.00%> (-10.05%) ⬇️
py-polars/polars/testing.py 92.65% <0.00%> (-0.57%) ⬇️
polars/polars-io/src/parquet/predicates.rs 71.18% <0.00%> (ø)
polars/polars-core/src/series/series_trait.rs 13.12% <0.00%> (ø)
polars/polars-lazy/src/lib.rs 100.00% <0.00%> (ø)
polars/polars-lazy/src/logical_plan/iterator.rs 98.02% <0.00%> (+0.24%) ⬆️
... and 209 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d248d00...5b0e5e2. Read the comment docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants