Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parquet: low memory arg #4050

Merged
merged 1 commit into from
Jul 17, 2022
Merged

parquet: low memory arg #4050

merged 1 commit into from
Jul 17, 2022

Conversation

ritchie46
Copy link
Member

@ritchie46 ritchie46 commented Jul 17, 2022

Added a low_memory argument to read/scan_parquet.

Currently this forces a slice to realloc. Really reducing memory pressure when we slice a small fraction of the complete array. This is at the expense of performance but when you are low on memory this might be worth it.

#3972

#3971

@github-actions github-actions bot added python Related to Python Polars rust Related to Rust Polars labels Jul 17, 2022
@codecov-commenter
Copy link

codecov-commenter commented Jul 17, 2022

Codecov Report

Merging #4050 (fc86a3f) into master (396018b) will decrease coverage by 0.00%.
The diff coverage is 57.44%.

@@            Coverage Diff             @@
##           master    #4050      +/-   ##
==========================================
- Coverage   62.82%   62.82%   -0.01%     
==========================================
  Files         449      449              
  Lines       74440    74544     +104     
==========================================
+ Hits        46764    46829      +65     
- Misses      27676    27715      +39     
Impacted Files Coverage Δ
.../polars-core/src/series/implementations/struct_.rs 56.55% <0.00%> (-1.19%) ⬇️
polars/polars-lazy/src/logical_plan/options.rs 56.75% <ø> (ø)
py-polars/polars/internals/frame.py 93.47% <ø> (+0.03%) ⬆️
py-polars/polars/internals/lazy_frame.py 76.00% <ø> (ø)
py-polars/polars/io.py 67.01% <ø> (ø)
polars/polars-core/src/frame/mod.rs 62.20% <14.28%> (-0.64%) ⬇️
polars/polars-io/src/parquet/read_impl.rs 89.65% <75.00%> (-0.31%) ⬇️
polars/polars-lazy/src/frame/parquet.rs 65.11% <80.00%> (+0.11%) ⬆️
polars/polars-io/src/parquet/read.rs 98.05% <100.00%> (+0.14%) ⬆️
polars/polars-lazy/src/logical_plan/builder.rs 90.29% <100.00%> (+0.04%) ⬆️
... and 30 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 396018b...fc86a3f. Read the comment docs.

@ritchie46 ritchie46 merged commit 8406450 into master Jul 17, 2022
@ritchie46 ritchie46 deleted the parquet branch July 17, 2022 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants