Update to arrow/parquet 55.2.0 #16575
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Update to released version
alamb
commented
Jun 26, 2025
| 01)CoalesceBatchesExec: target_batch_size=8192 | ||
| 02)--FilterExec: column1@0 != 42 | ||
| 03)----DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..137], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:137..274], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:274..411], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:411..547]]}, projection=[column1], file_type=parquet, predicate=column1@0 != 42, pruning_predicate=column1_null_count@2 != row_count@3 AND (column1_min@0 != 42 OR 42 != column1_max@1), required_guarantees=[column1 not in (42)] | ||
| 03)----DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..141], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:141..282], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:282..423], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:423..563]]}, projection=[column1], file_type=parquet, predicate=column1@0 != 42, pruning_predicate=column1_null_count@2 != row_count@3 AND (column1_min@0 != 42 OR 42 != column1_max@1), required_guarantees=[column1 not in (42)] |
There was a problem hiding this comment.
Some files can get (slightly) larger due to
alamb
commented
Jun 26, 2025
| 07)│ DataSourceExec │ | ||
| 08)│ -------------------- │ | ||
| 09)│ bytes: 1072 │ | ||
| 09)│ bytes: 1040 │ |
There was a problem hiding this comment.
I believe these files are now slightly smaller due to
This was referenced Jun 26, 2025
|
🤖 |
|
🤖: Benchmark completed Details
|
|
🤖 |
|
🤖: Benchmark completed Details
|
|
🤖 |
|
🤖: Benchmark completed Details
|
zhuqi-lucas
approved these changes
Jun 27, 2025
There was a problem hiding this comment.
LGTM, thanks @alamb , there are some performance regression for clickbench, but i may be noise?
Hmm looks non trivial -- I will rerun |
This comment was marked as outdated.
This comment was marked as outdated.
|
🤖 |
|
🤖: Benchmark completed Details
|
|
🤖 |
|
🤖: Benchmark completed Details
|
|
🤖 |
|
Hmm, next up will be to try and reproduce the slowdown locally 🤔 |
|
🤖: Benchmark completed Details
|
|
I am preparing to compare locally with 3649dc8 (the (venv) andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion$ git checkout `git merge-base alamb/alamb/arrow_55.2.0_upgrade_real apache/main`
Note: switching to '3649dc80cef2cdcd2df750e90e6ffd94ae6ae808'.
....
cargo build --profile=profiling -p datafusion-cli && cp target/profiling/datafusion-cli ~/Downloads/datafusion-cli-arrow-55.1.0 |
|
I couldn't really reproduce anything locally but the variation is pretty noisy with q21 When looking at that query however, I found some other performance improvement: |
|
I also don't really see any difference with q4 (another that was reported 10% slower) (venv) andrewlamb@Andrews-MacBook-Pro-2:~/Downloads$ ./datafusion-cli-alamb_arrow_55.2.0_upgrade_real -f q4-times-10.sql | grep Elapsed
Elapsed 0.312 seconds.
Elapsed 0.271 seconds.
Elapsed 0.266 seconds.
Elapsed 0.265 seconds.
Elapsed 0.273 seconds.
Elapsed 0.277 seconds.
Elapsed 0.267 seconds.
Elapsed 0.273 seconds.
Elapsed 0.269 seconds.
Elapsed 0.276 seconds.
(venv) andrewlamb@Andrews-MacBook-Pro-2:~/Downloads$ ./datafusion-cli-arrow-55.1.0 -f q4-times-10.sql |grep Elapsed
Elapsed 0.324 seconds.
Elapsed 0.289 seconds.
Elapsed 0.266 seconds.
Elapsed 0.269 seconds.
Elapsed 0.263 seconds.
Elapsed 0.272 seconds.
Elapsed 0.267 seconds.
Elapsed 0.275 seconds.
Elapsed 0.268 seconds.
Elapsed 0.262 seconds.I will next see if I can reproduce the same thing on the actual benchmark machine |
|
Here are my benchmark results on the same benchmark machine: q4: q21 (basically no change ...) So I am stumped |
|
An update here is I am working on some scripts to start measuring performance over time of datafusion so we can get a better handle on how performance is changing over time |
This is great for us to monitor performance! |
|
🤖 |
|
🤖: Benchmark completed Details
|
|
🤖 |
|
🤖: Benchmark completed Details
|
|
I am going to merge this PR in to unblock other things (like the PR for external parquet indexes) I am not really sure about the performance impact as I see some non trivial fluctuation. However, I have been working on a system to record and report DataFusion performance overtime. See #5504 for more detail or get a sneak peek here: https://github.com/alamb/datafusion-benchmarking If we have lost any performance let's plan to get it back over time |
|
🚀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
55.2.2#16492Rationale for this change
I would like to use new features in this minor version of arrow
What changes are included in this PR?
Are these changes tested?
By existing CI tests
Are there any user-facing changes?
No