Update cranelift requirement from 0.88.0 to 0.89.0 #123

dependabot · 2022-10-21T02:49:08Z

Updates the requirements on cranelift to permit the latest version.

Commits

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR
@dependabot recreate will recreate this PR, overwriting any edits that have been made to it
@dependabot merge will merge this PR after your CI passes on it
@dependabot squash and merge will squash and merge this PR after your CI passes on it
@dependabot cancel merge will cancel a previously requested merge and block automerging
@dependabot reopen will reopen this PR if it is closed
@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

Updates the requirements on [cranelift](https://github.com/bytecodealliance/wasmtime) to permit the latest version. - [Release notes](https://github.com/bytecodealliance/wasmtime/releases) - [Changelog](https://github.com/bytecodealliance/wasmtime/blob/main/docs/WASI-some-possible-changes.md) - [Commits](https://github.com/bytecodealliance/wasmtime/commits) --- updated-dependencies: - dependency-name: cranelift dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>

dependabot · 2022-10-21T02:49:09Z

The following labels could not be found: auto-dependencies.

dependabot · 2022-10-26T01:10:06Z

Looks like cranelift is up-to-date now, so this is no longer needed.

## Which issue does this PR close?  - Closes #. Related to apache#18084 ## Rationale for this change Run extended suite on PRs for critical areas, to avoid post merge bugfixing  ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?   --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

## Which issue does this PR close?  - Closes #. Followup on apache#18063 (review) ## Rationale for this change Use cheaper `NullBuffer::union` to apply null mask instead of iterator approach  ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?

## Which issue does this PR close?  - Closes #. Followup on apache#18063 (review) ## Rationale for this change Use cheaper `NullBuffer::union` to apply null mask instead of iterator approach  ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?   (cherry picked from commit 337378a)

…unctions in proto (apache#18024) ## Which issue does this PR close?  - Closes apache#17417. ## Rationale for this change  - Support `null_treatment`, `distinct`, and `filter` for window function in proto. - Support `null_treatment` for aggregate udf in proto. ## What changes are included in this PR?  - [x] Add `null_treatment`, `distinct`, `filter` fields to `WindowExprNode` message and handle them in `to/from_proto.rs`. - [x] Add `null_treatment` field to `AggregateUDFExprNode` message and handle them in `to/from_proto.rs`. - [ ] Docs update: I'm not sure where to add docs as declared in the issue description. ## Are these changes tested?  - Add tests to `roundtrip_window` for respectnulls, ignorenulls, distinct, filter. - Add tests to `roundtrip_aggregate_udf` for respectnulls, ignorenulls. ## Are there any user-facing changes?   N/A --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>

@dariocurr

## Which issue does this PR close?  - Doesn't close an issue. ## Rationale for this change  Hi we are hiop, a Serverless Data Logistic Platform. We use DataFusion as a core part of our backend engine, and it plays a crucial role in our data infrastructure. Our team members are passionate about the project and actively try contribute to its development (@dariocurr). We’d love to have Hiop listed among the Known Users to show our support and help the DataFusion community continue to grow. ## What changes are included in this PR?  Just adding hiop as known user ## Are these changes tested?  ## Are there any user-facing changes?

…8117) ## Which issue does this PR close?  - Closes apache#3695 - Closes apache#3797 ## Rationale for this change  Was looking at above issues and I don't believe we skip the failed rules for any tests anymore (default for the config is also `false`), apart from this cleanup, so filing this PR so we can close the issues. Seems we only do in this `window.slt` test after this fix: https://github.com/apache/datafusion/blob/621a24978a7a9c6d2b27973d1853dbc8776a56b5/datafusion/sqllogictest/test_files/window.slt#L2587-L2611 Which seems intentional. ## What changes are included in this PR?  Remove unnecessary `skip_failed_rules` config. ## Are these changes tested?  Existing tests. ## Are there any user-facing changes?  No.

## Which issue does this PR close?  - Closes #. ## Rationale for this change  `EXPLAIN ANALYZE` can be used for profiling and displays the results alongside the EXPLAIN plan. The issue is that it currently shows too many low-level details. It would provide a better user experience if only the most commonly used metrics were shown by default, with more detailed metrics available through specific configuration options. ### Example In `datafusion-cli`: ``` > CREATE EXTERNAL TABLE IF NOT EXISTS lineitem STORED AS parquet LOCATION '/Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem'; 0 row(s) fetched. Elapsed 0.000 seconds. explain analyze select * from lineitem where l_orderkey = 3000000; ``` The parquet reader includes a large number of low-level details: ``` metrics=[output_rows=19813, elapsed_compute=14ns, batches_split=0, bytes_scanned=2147308, file_open_errors=0, file_scan_errors=0, files_ranges_pruned_statistics=18, num_predicate_creation_errors=0, page_index_rows_matched=19813, page_index_rows_pruned=729088, predicate_cache_inner_records=0, predicate_cache_records=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_matched_bloom_filter=0, row_groups_matched_statistics=1, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, bloom_filter_eval_time=21.997µs, metadata_load_time=273.83µs, page_index_eval_time=29.915µs, row_pushdown_eval_time=42ns, statistics_eval_time=76.248µs, time_elapsed_opening=4.02146ms, time_elapsed_processing=24.787461ms, time_elapsed_scanning_total=24.17671ms, time_elapsed_scanning_until_data=23.103665ms] ``` I believe only a subset of it is commonly used, for example `output_rows`, `metadata_load_time`, and how many file/row-group/pages are pruned, and it would better to only display the most common ones by default. ### Existing `VERBOSE` keyword There is a existing verbose keyword in `EXPLAIN ANALYZE VERBOSE`, however it's turning on per-partition metrics instead of controlling detail level. I think it would be hard to mix this partition control and the detail level introduced in this PR, so they're separated: the following config will be used for detail level and the semantics of `EXPLAIN ANALYZE VERBOSE` keep unchanged. ### This PR: configurable explain analyze level 1. Introduced a new config option `datafusion.explain.analyze_level`. When set to `dev` (default value), all existing metrics will be shown. If set to `summary`, only `BaselineMetrics` will be displayed (i.e. `output_rows` and `elapsed_compute`). Note now we only include `BaselineMetrics` for simplicity, in the follow-up PRs we can figure out what's the commonly used metrics for each operator, and add them to `summary` analyze level, finally set the `summary` analyze level to default. 2. Add a `MetricType` field associated with `Metric` for detail level or potentially category in the future. For different configurations, a certain `MetricType` set will be shown accordingly. #### Demo ``` -- continuing the above example > set datafusion.explain.analyze_level = summary; 0 row(s) fetched. Elapsed 0.000 seconds. > explain analyze select * from lineitem where l_orderkey = 3000000; +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | plan_type | plan | +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Plan with Metrics | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=5, elapsed_compute=25.339µs] | | | FilterExec: l_orderkey@0 = 3000000, metrics=[output_rows=5, elapsed_compute=81.221µs] | | | DataSourceExec: file_groups={14 groups: [[Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-0.parquet:0..11525426], [Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-0.parquet:11525426..20311205, Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-1.parquet:0..2739647], [Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-1.parquet:2739647..14265073], [Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-1.parquet:14265073..20193593, Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-2.parquet:0..5596906], [Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-2.parquet:5596906..17122332], ...]}, projection=[l_orderkey, l_partkey, l_suppkey, l_linenumber, l_quantity, l_extendedprice, l_discount, l_tax, l_returnflag, l_linestatus, l_shipdate, l_commitdate, l_receiptdate, l_shipinstruct, l_shipmode, l_comment], file_type=parquet, predicate=l_orderkey@0 = 3000000, pruning_predicate=l_orderkey_null_count@2 != row_count@3 AND l_orderkey_min@0 <= 3000000 AND 3000000 <= l_orderkey_max@1, required_guarantees=[l_orderkey in (3000000)], metrics=[output_rows=19813, elapsed_compute=14ns] | | | | +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row(s) fetched. Elapsed 0.025 seconds. ``` Only `BaselineMetrics` are shown. ## What changes are included in this PR?  ## Are these changes tested?  UT ## Are there any user-facing changes? No   --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

…e#18091) ## Which issue does this PR close?  N/A ## Rationale for this change  There's a few functions in `datafusion/expr-common/src/type_coercion/aggregates.rs` that are unused elsewhere in the codebase, likely a remnant before the refactor to UDF, so removing them. Some are still used (`coerce_avg_type()` and `avg_return_type()`) so these are inlined into the Avg aggregate function (similar to Sum). Also refactor some window functions to use already available macros. ## What changes are included in this PR?  - Remove some unused functions - Inline avg coerce & return type logic - Refactor Spark Avg a bit to remove unnecessary code - Refactor ntile & nth window functions to use available macros ## Are these changes tested?  Existing tests. ## Are there any user-facing changes? Yes as these functions were publicly exported; however I'm not sure they were meant to be used by users anyway, given what they do.

apache#18099) ## Which issue does this PR close?  Fixes comparison errors when using dictionary-encoded types with comparison functions like NULLIF. ## Rationale for this change When using dictionary-encoded columns (e.g., Dictionary(Int32, Utf8)) in comparison operations with literals or other types, DataFusion would throw an error stating the types are not comparable. This was particularly problematic for functions like NULLIF which rely on comparison coercion. The issue was that comparison_coercion_numeric didn't handle dictionary types, even though the general comparison_coercion function did have dictionary support.  ## What changes are included in this PR? 1. Refactored dictionary comparison logic: Extracted common dictionary coercion logic into dictionary_comparison_coercion_generic to avoid code duplication. 2. Added numeric-specific dictionary coercion: Introduced dictionary_comparison_coercion_numeric that uses numeric-preferring comparison rules when dealing with dictionary value types. 3. Updated comparison_coercion_numeric: Added a call to dictionary_comparison_coercion_numeric in the coercion chain to properly handle dictionary types. 4. Added sqllogictest cases demonstrating the fix works for various dictionary comparison scenarios.  ## Are these changes tested? Yes, added tests in datafusion/sqllogictest/test_files/nullif.slt covering: - Dictionary type compared with string literal - String compared with dictionary type - Dictionary compared with dictionary All tests pass with the fix and would fail without it.  ## Are there any user-facing changes? This is a bug fix that enables previously failing queries to work correctly. No breaking changes or API modifications.

## Which issue does this PR close?  - Closes apache#18135 ## Rationale for this change  ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?   --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

## Which issue does this PR close?  - Closes #. ## Rationale for this change  ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?   --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

…pache#17478) (apache#18130) ## Which issue does this PR close?  - Related to apache#17405 - Related to apache#18072 ## Rationale for this change  See apache#17478 ## What changes are included in this PR?  See apache#17478 ## Are these changes tested?  ## Are there any user-facing changes?   Co-authored-by: Filippo Rossi <12383260+notfilippo@users.noreply.github.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

…nics (apache#18013) (apache#18131) ## Which issue does this PR close?  - Related to apache/datafusion-comet#2539 - Related to apache#18072 ## Rationale for this change  Return errors rather than panicking. ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?   Co-authored-by: Andy Grove <agrove@apache.org> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

…amps (apache#17777) (apache#18129) ## Which issue does this PR close?  - Related to apache#17776 - Related to apache#18072 ## Rationale for this change  If I've understood semantic equality correctly, any two timestamps should meet the bar for equality regardless of time units and timezones, but the current code doesn't reflect that. ## What changes are included in this PR?  Adds a branch to this method for timestamps. ## Are these changes tested?  Yes ## Are there any user-facing changes?  Yes  --------- Co-authored-by: Shiv Bhatia <shivbhatia10@gmail.com> Co-authored-by: Shiv Bhatia <sbhatia@palantir.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

…e#18161) (apache#18179) ## Which issue does this PR close?  - Related to apache#18070 - Part of apache#18072 ## Rationale for this change Fix performance regression in Datafusion 50 ## What changes are included in this PR? Backport apache#18161 to `branch-50` ## Are these changes tested? Yes ## Are there any user-facing changes? Fix performance regression Co-authored-by: Yongting You <2010youy01@gmail.com>

## Which issue does this PR close?  - Closes apache#16678. ## Rationale for this change  The issue has been fixed in apache#16639, this PR just adds a testcase for it. ## What changes are included in this PR?  Add a test case for `to_timestamp(double)` with vectorized input. Similar to the one presented in the issue. ## Are these changes tested?  Yes ## Are there any user-facing changes?   No

## Which issue does this PR close?  - Closes apache#18070 ## Rationale for this change  See the above issue and its comment apache#18070 (comment) ## What changes are included in this PR?  In nested loop join, when the join column includes `List(Utf8View)`, use `take()` instead of `to_array_of_size()` to avoid deep copying the utf8 buffers inside `Utf8View` array. This is the quick fix, avoiding deep copy inside `to_array_of_size()` is a bit tricky. Here is `ListArray`'s physical layout: https://arrow.apache.org/rust/arrow/array/struct.GenericListArray.html If multiple elements is pointing to the same list range, the underlying payload can't be reused.So the potential fix in `to_array_of_size` can only avoids copying the inner-inner utf8view array buffers, but can't avoid copying the inner array (i.e. views are still copied), and deep copying for other primitive types also can't be avoided. Seems this can be better solved when `ListView` type is ready 🤔 ### Benchmark I tried query 1 in apache#18070, but only used 3 randomly sampled `places` parquet file. 49.0.0: 4s 50.0.0: stuck > 1 minute PR: 4s Now the performance are similar, I suspect the most time is spend evaluating the expensive `array_has` so the optimization in apache#16996 can't help much. ## Are these changes tested?  Existing tests ## Are there any user-facing changes?  No  --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

## Which issue does this PR close?  - Closes apache#17913 . ## Rationale for this change  - Improve SQL code block rendering by upgrading `pydata-sphinx-theme` - fix sidebar layout ## What changes are included in this PR?  ## Are these changes tested?  yes ## Are there any user-facing changes? documentation ui

## Which issue does this PR close?  - part of apache#17427 ## Rationale for this change Adds regular joins (left, right, full, inner) for PWMJ as they behave differently in the code path.  ## What changes are included in this PR? Adds classic join + physical planner  ## Are these changes tested? Yes SLT tests + unit tests ## Follow up work to this pull request - Handling partitioned queries and multiple record batches (fuzz testing will be handled with this) - Simplify physical planning - Add more unit tests for different types (another pr as the LOC in this pr is getting a little daunting) next would be to implement the existence joins --------- Co-authored-by: Yongting You <2010youy01@gmail.com>

## Which issue does this PR close?  - Closes apache#17854 . ## Rationale for this change  ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?

…pache#18017) ## Which issue does this PR close?  - Closes apache#17993 ## Rationale for this change ``` DataFusion CLI v50.1.0 > SET TIME ZONE = '+08:00'; 0 row(s) fetched. Elapsed 0.011 seconds. > SELECT arrow_typeof(now()); +---------------------------------------+ | arrow_typeof(now()) | +---------------------------------------+ | Timestamp(Nanosecond, Some("+08:00")) | +---------------------------------------+ 1 row(s) fetched. Elapsed 0.015 seconds. > SELECT count(1) result FROM (SELECT now() as n) a WHERE n > '2000-01-01'::date; +--------+ | result | +--------+ | 1 | +--------+ 1 row(s) fetched. Elapsed 0.029 seconds. ```  ## What changes are included in this PR? When the timezone changes, re-register `now()` function  ## Are these changes tested?  ## Are there any user-facing changes?   --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

## Which issue does this PR close?  - Closes #. ## Rationale for this change According to three-valued logic we should return `null` and that's also what happens when the argument is not a constant as seen in the test.  ## What changes are included in this PR? Updated `ArrayHas::simplify` to explicitly handle `null`  ## Are these changes tested? Updated the `array_has` SQL test and added unit tests  ## Are there any user-facing changes? Yes, a minor change in behaviour wrt `null`

## Which issue does this PR close?  - Closes apache#16820 . ## Rationale for this change  ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?

## Which issue does this PR close?  - Part of apache#16602 ## Rationale for this change  Now we have to search in the code comment (or even implementation) to find the documentation of certain metrics, it would be better to open a page in the `user-guide` for metrics. The doc has to be manually updated, the metrics construction is scattered in the codebase, so it's hard to make it auto-generated. This PR only includes 2 common metrics, I plan to add more operator-specific metrics while working on apache#18116 ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?

## Which issue does this PR close? This PR is for consolidating all the `custom_data_source` examples into a single example binary. We are agreed on the pattern and we can apply it to the remaining examples  - part of #apache#18142. ## Rationale for this change  ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?   --------- Co-authored-by: Sergey Zhukov <szhukov@aligntech.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

…18491) ## Which issue does this PR close?  - Closes apache#17027 ## Rationale for this change  `output_batches` should be a common metric in all operators, thus should ideally be added to `BaselineMetrics` ``` > explain analyze select * from generate_series(1, 1000000) as t1(v1) order by v1 desc; +-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | plan_type | plan | +-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Plan with Metrics | SortExec: expr=[v1@0 DESC], preserve_partitioning=[false], metrics=[output_rows=1000000, elapsed_compute=535.320324ms, output_bytes=7.6 MB, output_batches=123, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0] | | | ProjectionExec: expr=[value@0 as v1], metrics=[output_rows=1000000, elapsed_compute=208.379µs, output_bytes=7.7 MB, output_batches=123] | | | LazyMemoryExec: partitions=1, batch_generators=[generate_series: start=1, end=1000000, batch_size=8192], metrics=[output_rows=1000000, elapsed_compute=15.924291ms, output_bytes=7.7 MB, output_batches=123] | | | | +-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row(s) fetched. Elapsed 0.492 second ``` ## What changes are included in this PR?  - Added `output_batches` into `BaselineMetrics` with `DEV` MetricType - Tracked through `record_poll()` API - Changes are similar to apache#18268 - Refactored `assert_metrics` macro to take multiple metrics strings for substring check - Added `output_bytes` and `output_batches` tracking in `TopK` operator - Added `baseline` metrics for `RepartitionExec` ## Are these changes tested?  Added UT ## Are there any user-facing changes?   Changes in the `EXPLAIN ANALYZE` output, `output_batches` will be added to `metrics=[...]`

## Which issue does this PR close?  - Closes apache#16688. ## Rationale for this change  Currently Datafusion can only read Arrow files if the're in the File format, not the Stream format. I work with a bunch of Stream format files and wanted native support. ## What changes are included in this PR?  To accomplish the above, this PR splits the Arrow datasource into two separate implementations (`ArrowStream*` and `ArrowFile*`) with a facade on top to differentiate between the formats at query planning time. ## Are these changes tested?  Yes, there are end-to-end sqllogictests along with tests for the changes within datasource-arrow. ## Are there any user-facing changes?   Technically yes, in that we support a new format now. I'm not sure which documentation would need to be updated? --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

## Which issue does this PR close?  - Closes #. ## Rationale for this change `cargo-machete` identifies an unused dependency and this blocks a bunch of dependabot updates PRs apache#18580  ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?

… require code changes. (apache#18586) ## Which issue does this PR close?  Part of apache#18503 ## Rationale for this change  Enforce the lint rule to all crates that are already passing this extra check, and we don't need further code change on them. ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?   Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

## Which issue does this PR close?  - Closes apache#18341. - Closes apache#9370 ## Rationale for this change  Cases where two RepartitionExec operators appear consecutively in the plan. This is unneeded overhead that eliminating provides speed ups. Full Report: [The Physical Optimizer and Fixing Consecutive Repartitions In the Enforce Distribution Rule.pdf](https://github.com/user-attachments/files/23420831/The.Physical.Optimizer.and.Fixing.Consecutive.Repartitions.In.the.Enforce.Distribution.Rule.pdf) Issue Report: [Fixing Consecutive Repartitions In the Enforce Distribution Rule.pdf](https://github.com/user-attachments/files/23420880/Fixing.Consecutive.Repartitions.In.the.Enforce.Distribution.Rule.pdf) ## What changes are included in this PR?  Change to repartition adding logic in `enforce_distribution.rs` A ton of test and bench updates to mirror new behavior ## Are these changes tested?  Yes benchmarked and tested, check report for benchmarks ## Are there any user-facing changes?   --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

…8540) ## Which issue does this PR close?  - Closes apache#18155. ## Rationale for this change  ## What changes are included in this PR?  Merges the functionality of `CoalesceAsyncExecInput` into `CoalesceBatches` to remove redundant optimizer logic and simplify batch coalescing behavior. ## Are these changes tested?  Behavior is covered by existing ``CoalesceBatches and optimizer tests. ## Are there any user-facing changes? No

…e#18603) ## Which issue does this PR close?  - Closes apache#18375 . ## What changes are included in this PR? Enhance the help message for invalid command, add the `help` command in the help message.  ## Are these changes tested? Manual test <img width="737" height="190" alt="image" src="https://github.com/user-attachments/assets/073d4b96-db6f-448b-a7a4-3f9fa456a72e" />  ## Are there any user-facing changes? No

## Which issue does this PR close?  - Closes apache#18597 ## Rationale for this change  A check is recently added to `invoke_with_args` that checks for the output type of the result with the expected output type from the UDF - apache#17515. Because the fast path misses adding the timezone, the assertion added in this PR fails. ## What changes are included in this PR?  Include timezone information in the fast path. ## Are these changes tested?  Yes, added a unit test ## Are there any user-facing changes?  No

## Which issue does this PR close?  - Closes: apache#18606 - Relates to: apache#7001 ## Rationale for this change Moe coalesce batches inside filter exec. We can use `BatchCoalescer ::push_batch_with_filter` which should give a speed up compared to filtering individual batches + concatenating afterwards.  ## What changes are included in this PR? Changes the FilterExec to coalesce batches inside. I did not make a change to remove CoalesceBatchesExec from all the plans, I plan to create an issue and a PR after this is merged to do so. Now it should be mostly a no-op with limited overhead as input batches are already well-sized.  ## Are these changes tested?  ## Are there any user-facing changes?

… SQL planner based on Datatype (apache#18599) ## Which issue does this PR close?  - as discussed in apache#17261 ## Rationale for this change Logical Plan for datatype Int64 and UInt64 differs, UInt64 Logical Plan's Union are wrapped up in Projection, and EliminateNestedUnion OptimezerRule is not applied leading to significantly longer execution time.  ## What changes are included in this PR? Separating Benchmarks based on datatype, converting a datatype specific function to a generic one.  ## Are these changes tested? Yes.  ## Are there any user-facing changes? No, benchmarks only.   --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

…pache#18614) ## Which issue does this PR close?  Part of apache#18613 ## Rationale for this change  See issue ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?

## Which issue does this PR close?  Currently ASF Infra team investigates the reason why changes in `.asf.yaml` are not effective for `branch-*`, I was asked to make a whitespace edit so ASF team can track the progress - Closes #. ## Rationale for this change  ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?

## Which issue does this PR close?  - Closes apache#18634 . ## Rationale for this change Adding a ansi flag to expand coverage for Spark built in functions, Spark 4.0 sets ansi mode as true by default. Currently the flag is planned to be used to `datafusion-spark` crate via ScalarConfigArgs, however it can also be used for DF if ansi mode is in the roadmap  ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?

## Which issue does this PR close?  - Closes apache#18330 . ## Rationale for this change  Reduce code duplication. ## What changes are included in this PR?  A util function replacing many calls which are using the same code. ## Are these changes tested?  No logic should change whatsoever, so each area which now uses this code should have it's own tests and benchmarks unmodified. ## Are there any user-facing changes?   Yes, there is now a new pub function. No other changes to API. --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

## Which issue does this PR close?  Part of apache#18092 ## Rationale for this change Use simpler coercion API. ## What changes are included in this PR?  Use coercion API for bit aggregate functions(`bit_and`, `bit_or`, `bit_xor`), instead of enumerating numeric types. Handle `Null` datatype explicitly in above functions. Introduce `NoopAccumulator` for easier handling of the above. ## Are these changes tested?  Existing tests. ## Are there any user-facing changes?  No.

…e#18450) ## Which issue does this PR close?  - Closes apache#17887. ## Rationale for this change  See linked issue above ## What changes are included in this PR?  - Add validation when planning `CreateFunction` - Enforce consistent parameter style (positional or named) - Non-default params cannot follow default params - `CreateFunction` parameter names are now preserved from the parse tree - If we encounter a named parameter when constructing a `Placeholder` - We try to rewrite this to a positional parameter from the available param types - If no matching param type is found, report an error - Update `ScalarFunctionWrapper` to handle defaults - Preserve the parsed defaults - Generate all valid signatures for all possible combinations of arguments - Fall back to default expr when no matching argument is provided ## Are these changes tested?  Yes, see added / adjusted unit tests ## Are there any user-facing changes?  Yes, if the approach here is acceptable we should update the SQL UDF examples in `datafusion-examples/examples/function_factory.rs` (TODO). Also, note that due to ambiguity between `PREPARE` and `CREATE FUNCTION` param context one error message now has reduced fidelity. > Invalid placeholder, not a number: $foo This can now be triggered either by using named params in a prepared statement or when referencing an undefined named param in a SQL UDF. New message: > Unknown placeholder: $foo Finally, there are two additional user-facing errors when planning `CreateFunction`: - When named / positional parameter styles are mixed - When non-default arguments follow default arguments

## Which issue does this PR close?  - part of #apache#18142. ## Rationale for this change This PR is for consolidating all the `data_io` examples (parquet, catalog, remote_catalog, json_shredding, query_http_csv) into a single example binary. We are agreed on the pattern and we can apply it to the remaining examples  ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?   --------- Co-authored-by: Sergey Zhukov <szhukov@aligntech.com>

## Which issue does this PR close?  - Closes apache#18145 ## Rationale for this change  Allows constructing `CachedParquetFileReader::new()` manually, allowing users to create it with a custom object reader (inner) object. ## What changes are included in this PR?  Adds a simple constructor for `CachedParquetFileReader` ## Are these changes tested?  Added a test ## Are there any user-facing changes?  Yes, new constructor available

## Which issue does this PR close?  - part of apache#17558 ## Rationale for this change Let's update the version numbers! ## What changes are included in this PR? Update numbers and add changelog. I plan to merge this PR now and I will make a follow on PR (and forward port it to main) with final touchups ## Are these changes tested?  ## Are there any user-facing changes?

…che#18629) ## Which issue does this PR close?  - Closes apache#18597 ## Rationale for this change  A check is recently added to `invoke_with_args` that checks for the output type of the result with the expected output type from the UDF - apache#17515. Because the fast path misses adding the timezone, the assertion added in this PR fails. ## What changes are included in this PR?  Include timezone information in the fast path. ## Are these changes tested?  Yes, added a unit test and SLT test ## Are there any user-facing changes?  No  ## Which issue does this PR close?  - Closes #. ## Rationale for this change  ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?

## Which issue does this PR close?  - Closes #. ## Rationale for this change  Still tracking with ASF Infra, they suspect changes are not in effect because of non existent branches ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?

## Which issue does this PR close?  - Closes apache#17547 ## Rationale for this change  See the issue for the background. If the optimizer made the wrong join order decision, and put a very small input at the probe side of NLJ, the NLJ operator now can handle it much faster than before. For implementation, before it's always handling `(one_left_row X right_batch)` in the inner loop, this PR do join multiple left rows at once with the right batch, if the right batch is very small. The NLJ microbench result, only Q13 is for this workload: ``` ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓ ┃ Query ┃ before ┃ improve-nlj-small-right ┃ Change ┃ ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩ │ QQuery 1 │ 85.31 ms │ 85.75 ms │ no change │ │ QQuery 2 │ 111.36 ms │ 109.88 ms │ no change │ │ QQuery 3 │ 180.99 ms │ 181.56 ms │ no change │ │ QQuery 4 │ 340.38 ms │ 355.24 ms │ no change │ │ QQuery 5 │ 248.62 ms │ 231.90 ms │ +1.07x faster │ │ QQuery 6 │ 1680.89 ms │ 1682.07 ms │ no change │ │ QQuery 7 │ 233.65 ms │ 234.83 ms │ no change │ │ QQuery 8 │ 1679.12 ms │ 1675.63 ms │ no change │ │ QQuery 9 │ 266.52 ms │ 266.54 ms │ no change │ │ QQuery 10 │ 544.66 ms │ 544.71 ms │ no change │ │ QQuery 11 │ 274.43 ms │ 265.71 ms │ no change │ │ QQuery 12 │ 275.11 ms │ 274.72 ms │ no change │ │ QQuery 13 │ 76.56 ms │ 1.88 ms │ +40.70x faster │ └──────────────┴────────────┴─────────────────────────┴────────────────┘ ``` In DF49 it's around 4ms. ## What changes are included in this PR?  - Added one microbench query targeting small right input workload - Added one branch in the NLj's right input handling logic: if the current right batch is very small, try to join it with multiple left rows. ## Are these changes tested?  This can be covered by existing tests: this additional path is not only triggered if the entire right input is small. For regular workloads, the final input batch can be also very small, so this new path can be triggered and tested. ## Are there any user-facing changes?

## Which issue does this PR close?  Part of apache#12725 ## Rationale for this change  Prefer to avoid user_defined for consistency in function definitions. ## What changes are included in this PR?  Refactor signature of Spark bit shift functions (left, right, right unsigned) to use coercion API instead of being user defined. Also refactor the bit shift code to have a common base struct. Move the Rust unit tests to SLTs. ## Are these changes tested?  Existing tests. ## Are there any user-facing changes?  No.

…o `false` (apache#18441) ## Which issue does this PR close?  - Part of apache#9924 ## Rationale for this change  Setting a default of `true` is too permissive; we can see it allows specifying it on `median` for example even though that function doesn't take the config into account. It seems only `array_agg`, `first_value` and `last_value` actually respect the config this setting handles so it makes more sense to make this false by default. ## What changes are included in this PR?  Change default of `AggregateUDFImpl::supports_null_handling_clause` to `false` (from `true`). Adjust `array_agg`, `first_value` and `last_value` to implement it as `true`. ## Are these changes tested?  Existing tests, also added a negative case for `median` (expect SQL parsing to fail if providing null handling clause). ## Are there any user-facing changes?  Behaviour change as default of a trait method is changing. Added section to upgrade guide.

## Which issue does this PR close? Test was updated in https://github.com/apache/datafusion/pull/18521/files#diff-fe22c0d1093b8b848a9317b0796f58276d36b98c8444865dcc0c252fd5b02632R102 but docs don't reflect the new plan.  - Closes #. ## Rationale for this change  ## What changes are included in this PR? Only doc changes for tests.  ## Are these changes tested?  ## Are there any user-facing changes? No   Co-authored-by: Xander <xbailey@palantir.com>

…ion-catalog` (apache#18638) ## Which issue does this PR close?  - Closes apache#18609. ## Rationale for this change  The type `impl AsRef<str>` is implicitly `Sized`, meaning its size is known at compile time. `&(impl AsRef<str>)` is a reference to a concrete type that implements `AsRef<str>` and is `Sized`. However, this `Sized` bound prevents passing references to dynamically sized types such as `&str`, `String`, or `Box<dyn Trait>`. Therefore, we add `+ ?Sized` to indicate that the referenced type might not have a fixed size — that is, it allows the referenced type to be dynamically sized. ## What changes are included in this PR?  - Enforce lint rule `clippy::needless_pass_by_value` to `datafusion-catalog`. ## Are these changes tested?  ## Are there any user-facing changes?   Signed-off-by: Alan Tang <jmtangcs@gmail.com>

## Which issue does this PR close? - part of apache#17558 ## Rationale for this change - Bring some small changes to changelog to main ## What changes are included in this PR? Cherry-pick this PR to main: - apache#18592 ## Are these changes tested? By CI ## Are there any user-facing changes? New changelog content ## Which issue does this PR close?  - Closes #. ## Rationale for this change  ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?

…che#18348) ## Which issue does this PR close?  - Relates to apache#2406 ## Rationale for this change  Make it easier to write distinct variations of aggregate functions be refactoring some of the common code together; specifically how they handle maintaining the complete set of distinct primitive values, as this code was duplicated across different functions. ## What changes are included in this PR?  Introduce new `GenericDistinctBuffer` which has methods similar to `Accumulator` to manage an internal `HashSet` of values, so implementations like `percentile_cont` and `sum` can use it internally and only implement their own evaluate functions. ## Are these changes tested?  Existing tests. ## Are there any user-facing changes?  No.

…ion-datasource-avro` (apache#18641) ## Which issue does this PR close?  - Closes apache#18612. ## Rationale for this change  ## What changes are included in this PR?  - Enforce lint rule `clippy::needless_pass_by_value` to `datafusion-datasource-avro`. ## Are these changes tested?  ## Are there any user-facing changes?   Signed-off-by: Alan Tang <jmtangcs@gmail.com>

dependabot bot closed this Oct 26, 2022

dependabot bot deleted the dependabot/cargo/master/cranelift-0.89.0 branch October 26, 2022 01:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update cranelift requirement from 0.88.0 to 0.89.0 #123

Update cranelift requirement from 0.88.0 to 0.89.0 #123

Uh oh!

dependabot bot commented on behalf of github Oct 21, 2022 •

edited

Loading

Uh oh!

dependabot bot commented on behalf of github Oct 21, 2022

Uh oh!

dependabot bot commented on behalf of github Oct 26, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Update cranelift requirement from 0.88.0 to 0.89.0 #123

Update cranelift requirement from 0.88.0 to 0.89.0 #123

Uh oh!

Conversation

dependabot bot commented on behalf of github Oct 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dependabot bot commented on behalf of github Oct 21, 2022

Uh oh!

dependabot bot commented on behalf of github Oct 26, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

dependabot bot commented on behalf of github Oct 21, 2022 •

edited

Loading