Sort file names in a directory #2730 #2735

yourenawo · 2022-06-15T10:54:41Z

Sort file names in a directory
fixed: #2730

yourenawo

andygrove

Partitions are processed in parallel on multiple threads so DataFusion cannot provide any guarantee of ordering of results unless the query contains an ORDER BY clause.

However, sorting files by filename seems reasonable to me from a UX point of view so I am fine with this change.

codecov-commenter · 2022-06-15T15:41:31Z

Codecov Report

Merging #2735 (13991fc) into master (ef9df29) will increase coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2735      +/-   ##
==========================================
+ Coverage   84.89%   84.92%   +0.03%     
==========================================
  Files         270      270              
  Lines       47817    47915      +98     
==========================================
+ Hits        40593    40693     +100     
+ Misses       7224     7222       -2

Impacted Files	Coverage Δ
datafusion/data-access/src/object_store/local.rs	`89.03% <100.00%> (+1.91%)`	⬆️
datafusion/core/src/physical_plan/metrics/value.rs	`86.93% <0.00%> (-0.51%)`	⬇️
datafusion/expr/src/logical_plan/plan.rs	`73.91% <0.00%> (ø)`
datafusion/common/src/scalar.rs	`74.94% <0.00%> (+0.11%)`	⬆️
datafusion/optimizer/src/filter_push_down.rs	`98.32% <0.00%> (+0.16%)`	⬆️
datafusion/expr/src/expr_fn.rs	`89.41% <0.00%> (+1.17%)`	⬆️
datafusion/optimizer/src/utils.rs	`34.21% <0.00%> (+1.41%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef9df29...13991fc. Read the comment docs.

alamb

A test might be good to ensure this behavior doesn't revert in the future

@yourenawo there appears to be a CI failure https://github.com/apache/arrow-datafusion/runs/6902662999?check_suite_focus=true due to not running cargo fmt

Added a test case for sorting directories.

alamb

Thanks @yourenawo ! Looks great

* Update local.rs * Update local.rs * Update local.rs Added a test case for sorting directories.

Update local.rs

0540ef7

yourenawo changed the title ~~fix #2730~~ Sort file names in a directory #2730 Jun 15, 2022

yourenawo commented Jun 15, 2022

View reviewed changes

andygrove approved these changes Jun 15, 2022

View reviewed changes

alamb reviewed Jun 15, 2022

View reviewed changes

Update local.rs

01cac28

yourenawo requested a review from alamb June 16, 2022 01:16

Update local.rs

13991fc

Added a test case for sorting directories.

alamb approved these changes Jun 16, 2022

View reviewed changes

alamb merged commit 0e416f0 into apache:master Jun 16, 2022

waynexia pushed a commit to waynexia/arrow-datafusion that referenced this pull request Jun 20, 2022

Sort file names in a directory apache#2730 (apache#2735)

28352fd

* Update local.rs * Update local.rs * Update local.rs Added a test case for sorting directories.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sort file names in a directory #2730 #2735

Sort file names in a directory #2730 #2735

yourenawo commented Jun 15, 2022 •

edited

Loading

yourenawo left a comment

andygrove left a comment

codecov-commenter commented Jun 15, 2022 •

edited

Loading

alamb left a comment

alamb left a comment

Sort file names in a directory #2730 #2735

Sort file names in a directory #2730 #2735

Conversation

yourenawo commented Jun 15, 2022 • edited Loading

yourenawo left a comment

Choose a reason for hiding this comment

andygrove left a comment

Choose a reason for hiding this comment

codecov-commenter commented Jun 15, 2022 • edited Loading

Codecov Report

alamb left a comment

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

yourenawo commented Jun 15, 2022 •

edited

Loading

codecov-commenter commented Jun 15, 2022 •

edited

Loading