diff --git a/datafusion/CHANGELOG.md b/datafusion/CHANGELOG.md index 5b919a1f21d7..38fccecb8707 100644 --- a/datafusion/CHANGELOG.md +++ b/datafusion/CHANGELOG.md @@ -21,101 +21,144 @@ ## [20.0.0](https://github.com/apache/arrow-datafusion/tree/20.0.0) (2023-03-10) -**Merged PRs** - -Note that the changelog generator is failing due to https://github.com/apache/arrow-datafusion/issues/5549 so -the changelog for this release was produced manually by running `git log --oneline 19.0.0..main` and only shows -merged PRs. - -- 8c34ca4fa (origin/main, main) Add UserDefinedLogicalNodeCore (#5521) -- 7f84503bb revert accidently deleted size code in count_distinct (#5533) -- 6a8c4a601 fix: return schema of ExtensionPlan instead of its children's (#5514) -- 55c7c4d8e Minor: Move `ObjectStoreRegistry` to datafusion_execution crate (#5478) -- 07b07d52f Add dbbenchmark URL to db-benchmark readme (#5503) -- 9878ee011 minor: fix clippy problem in new version. (#5532) -- 8a1cb7e3c fixed hash_join tests after passing boolean by value (#5531) -- f5d23ff58 memory limited hash join (#5490) -- ff013e245 minor: improve error style (#5510) -- e46924d80 feat: add `arrow_cast` function to support supports arbitrary arrow types (#5166) -- 84530a2f5 build(deps): update sqlparser requirement from 0.30 to 0.32 w/ API update (#5457) -- 8a1b13390 Allow setting config extensions for TaskContext (#5497) -- 0ead640e8 Minor: Improve docs for UserDefinedLogicalNode `dyn_eq` and `dyn_hash` (#5515) -- 89ca58326 feat: interval add timestamp (#5491) -- 9e32de344 Pass booleans by value instead of by reference (#5487) -- deeaa5632 Minor: Move TableProviderFactories up out of RuntimeEnv and into SessionState (#5477) -- 50e9d78da feat: `ParquetExec` predicate preservation (#5495) -- d7b221213 feat: add optimization rules for bitwise operations (#5423) -- 928662bb1 chore: Remove references from SessionState from physical_plan (#5455) -- 8d8b765f4 Implement `Debug` for `ExecutionProps` and `VarProvider` (#5489) -- a6f4a7719 feat: Support bitwise operations for unsigned integer types (#5476) -- 99ef98931 Apply workaround for #5444 to DataFrame::describe (#5468) -- d0bd28eef feat: eliminate the duplicated sort keys in Order By clause (#5462) -- 21e33a3d1 Propagate timezone to created arrays (#5481) -- 978ec2dc8 refactor: make GeometricMean not to have update and merge (#5469) -- c56b94765 feat: add name() method to UserDefinedLogicalNode (#5450) -- eb9541380 Comment out description fields in issue templates (#5482) -- c37ddf72e feat: express unsigned literal in substrait (#5448) -- 53a638ef3 fix: build union schema with child has same column name but qualifier is different (#5452) -- 8d195a8c0 refactor: make sum_distinct not to have update and merge (#5474) -- 132ae42f4 Fix decimal dyn scalar op (#5465) -- e9852074b feat: `extensions_options` macro (#5442) -- ac618f034 enable hash joins on FixedSizeBinary columns (#5461) -- 8de7ea45e Fix is_distinct from for float NaN values (#5446) -- 61fc51446 Implement/fix Eq and Hash for Expr and LogicalPlan (#5421) -- be6efbc93 [feat]:fast check has column (#5328) -- 1455b025c Parquet sorting benchmark (#5433) -- d11820aa2 refactor count_distinct to not to have update and merge (#5408) -- ddd64e7ed build(deps): update zstd requirement from 0.11 to 0.12 (#5458) -- db4610e1d Upgrade bytes to 1.4 (#5460) -- 49473d68b add std,median result to describe method (#5445) -- a95e0ec2f minor: Port more window tests to sqlogictests (#5434) -- f68214dc6 Use compute_op_dyn_scalar for datatime (#5315) -- a4b47d8c8 add a unit test that cover cast bug. (#5443) -- 17b2f11b5 create new `datafusion-execution` crate, start splitting code out (#5432) -- f9f40bf70 minor: fix clippy in nightly. (#5440) -- 3c1e4c0fd Support for Sliding Windows Joins with Symmetric Hash Join (SHJ) (#5322) -- 03fbf9fec refactor: ParquetExec logical expr. => phys. expr. (#5419) -- c4b895822 Update README.md fix [DataFusion] links (#5438) -- 793feda23 add mean result for describe method (#5435) -- eef046441 add expr_fn::median (#5437) -- d076ab3b2 Bug/union wrong casting (#5342) -- 0000d4fce reimplement `push_down_projection` and `prune_column`. (#4465) -- 25b4f673d Add (#5409) -- d9669841c fix nested loop join with literal join filter (#5431) -- 96aa2a677 add a describe method on DataFrame like Polars (#5226) -- ea3b965dd Memory reservation & metrics for cross join (#5339) -- 20d08ab1f Optimize count_distinct.size (#5377) -- c477fc0ca Fix filter pushdown for extension plans (#5425) -- c676d1026 Also push down all filters in TableProvider (#5420) -- 06fecacb3 Update arrow 34 (#5375) -- 411807609 Parquet limit pushdown (#5404) (#5416) -- 58cd1bf0b Move file format config.rs to live with the rest of the datasource code (#5406) -- 8202a395a Support Zstd compressed files (#5397) -- 5ffa8d78f Add example of catalog API usage (#5291) (#5326) -- a7a824ad9 Add support for protobuf serialisation of Arrow Map type (#5359) -- cb09142c7 minor: port window tests to slt (part 2) (#5399) -- 69503ea0f fix(docs): fix typos (#5403) -- fad360df0 rebase (#5367) -- 38185cacd enhance: remove more projection. (#5402) -- 0b77ec27c refactor `push_down_filter` to fix dead-loop and use optimizer_recurse. (#5337) -- 8b92b9b6c feat: add eliminate_unnecessary_projection rule (#5366) -- 645428f59 minor: support forgotten large_utf8 (#5393) -- 248d6fc1b Minor: add tests for subquery to join (#5363) -- 85ed38673 bugfix: fix master `bors` problem. (#5395) -- a870e460d feat: rule ReplaceDistinctWithAggregate (#5354) -- 4d4c5678e chore: add known project ZincObserve (#5376) -- 1841736d9 refactor: parquet pruning simplifications (#5386) -- 722490121 Optimization of the function "intersect" (#5388) -- d64076b46 docs: clarify spark (#5391) -- 47bdda60b UDF zero params #5378 (#5380) -- 32a238c50 Added tests for "like_for_type_coercion" and "test_type_coercion_rewrite" (#5389) -- 9fd6efa51 minor: make table resolution an independent function ... (#5373) -- 05d8a2527 minor: port predicates tests to sqllogictests (#5374) -- 49237a21a Bug fix: Window frame range value outside the type range (#5384) -- d6ef46374 Fixed small typos in files of the optimizer (#5356) -- cef119da9 fix: misc phys. expression display bugs (#5387) -- ac33ebc8f Prepare for 19.0.0 release (#5381) -- 1309267e7 minor: disable tpcds-q41 due to not support decorrelate disjunction subquery. (#5369) +[Full Changelog](https://github.com/apache/arrow-datafusion/compare/19.0.0...20.0.0-rc1) + +**Breaking changes:** + +- Minor: Move TableProviderFactories up out of `RuntimeEnv` and into `SessionState` [#5477](https://github.com/apache/arrow-datafusion/pull/5477) (alamb) +- chore: Remove references from SessionState from physical_plan [#5455](https://github.com/apache/arrow-datafusion/pull/5455) (alamb) +- Implement `Debug` for `ExecutionProps` and `VarProvider` [#5489](https://github.com/apache/arrow-datafusion/pull/5489) (alamb) + +**Implemented enhancements:** + +- Add UserDefinedLogicalNodeCore [#5521](https://github.com/apache/arrow-datafusion/pull/5521) (mslapek) +- feat: add `arrow_cast` function to support supports arbitrary arrow types [#5166](https://github.com/apache/arrow-datafusion/pull/5166) (alamb) +- feat: interval add timestamp [#5491](https://github.com/apache/arrow-datafusion/pull/5491) (Weijun-H) +- feat: `ParquetExec` predicate preservation [#5495](https://github.com/apache/arrow-datafusion/pull/5495) (crepererum) +- feat: add optimization rules for bitwise operations [#5423](https://github.com/apache/arrow-datafusion/pull/5423) (izveigor) +- feat: Support bitwise operations for unsigned integer types [#5476](https://github.com/apache/arrow-datafusion/pull/5476) (izveigor) +- feat: eliminate the duplicated sort keys in Order By clause [#5462](https://github.com/apache/arrow-datafusion/pull/5462) (jackwener) +- feat: add name() method to UserDefinedLogicalNode [#5450](https://github.com/apache/arrow-datafusion/pull/5450) (waynexia) +- feat: express unsigned literal in substrait [#5448](https://github.com/apache/arrow-datafusion/pull/5448) (waynexia) +- feat: `extensions_options` macro [#5442](https://github.com/apache/arrow-datafusion/pull/5442) (crepererum) +- [feat]:fast check has column [#5328](https://github.com/apache/arrow-datafusion/pull/5328) (suxiaogang223) +- feat: eliminate unnecessary projection. [#5366](https://github.com/apache/arrow-datafusion/pull/5366) (jackwener) + +**Fixed bugs:** + +- revert accidently deleted size code in count_distinct [#5533](https://github.com/apache/arrow-datafusion/pull/5533) (comphead) +- fix: return schema of ExtensionPlan instead of its children's [#5514](https://github.com/apache/arrow-datafusion/pull/5514) (waynexia) +- fix: logical merge conflict -- hash_join tests with passing boolean by value [#5531](https://github.com/apache/arrow-datafusion/pull/5531) (korowa) +- fix: build union schema with child has same column name but qualifier… [#5452](https://github.com/apache/arrow-datafusion/pull/5452) (yukkit) +- Fix is_distinct from for float NaN values [#5446](https://github.com/apache/arrow-datafusion/pull/5446) (comphead) +- Bug/union wrong casting [#5342](https://github.com/apache/arrow-datafusion/pull/5342) (berkaysynnada) +- fix nested loop join with literal join filter [#5431](https://github.com/apache/arrow-datafusion/pull/5431) (ygf11) +- Fix filter pushdown for extension plans [#5425](https://github.com/apache/arrow-datafusion/pull/5425) (thinkharderdev) +- Bug fix: Window frame range value outside the type range [#5384](https://github.com/apache/arrow-datafusion/pull/5384) (mustafasrepo) +- fix: misc phys. expression display bugs [#5387](https://github.com/apache/arrow-datafusion/pull/5387) (crepererum) + +**Documentation updates:** + +- Minor: Improve docs for UserDefinedLogicalNode `dyn_eq` and `dyn_hash` [#5515](https://github.com/apache/arrow-datafusion/pull/5515) (alamb) +- chore: add known project ZincObserve [#5376](https://github.com/apache/arrow-datafusion/pull/5376) (hengfeiyang) +- docs: clarify spark [#5391](https://github.com/apache/arrow-datafusion/pull/5391) (hyoklee) + +**Merged pull requests:** + +- Manual changelog for 20.0.0 [#5551](https://github.com/apache/arrow-datafusion/pull/5551) (andygrove) +- Prepare for 20.0.0 release [Part 1] [#5539](https://github.com/apache/arrow-datafusion/pull/5539) (andygrove) +- chore: deduplicate workspace fields in Cargo.toml [#5519](https://github.com/apache/arrow-datafusion/pull/5519) (waynexia) +- Add necessary features to optimizer [#5540](https://github.com/apache/arrow-datafusion/pull/5540) (viirya) +- Minor: add the concise way for matching numerics [#5537](https://github.com/apache/arrow-datafusion/pull/5537) (izveigor) +- Add UserDefinedLogicalNodeCore [#5521](https://github.com/apache/arrow-datafusion/pull/5521) (mslapek) +- revert accidently deleted size code in count_distinct [#5533](https://github.com/apache/arrow-datafusion/pull/5533) (comphead) +- fix: return schema of ExtensionPlan instead of its children's [#5514](https://github.com/apache/arrow-datafusion/pull/5514) (waynexia) +- Minor: Move `ObjectStoreRegistry` to datafusion_execution crate [#5478](https://github.com/apache/arrow-datafusion/pull/5478) (alamb) +- Minor: Add db-benchmark URL to db-benchmark readme [#5503](https://github.com/apache/arrow-datafusion/pull/5503) (alamb) +- minor: fix clippy problem in new version. [#5532](https://github.com/apache/arrow-datafusion/pull/5532) (jackwener) +- fix: logical merge conflict -- hash_join tests with passing boolean by value [#5531](https://github.com/apache/arrow-datafusion/pull/5531) (korowa) +- Memory limited hash join [#5490](https://github.com/apache/arrow-datafusion/pull/5490) (korowa) +- minor: improve error style [#5510](https://github.com/apache/arrow-datafusion/pull/5510) (alamb) +- feat: add `arrow_cast` function to support supports arbitrary arrow types [#5166](https://github.com/apache/arrow-datafusion/pull/5166) (alamb) +- build(deps): update sqlparser requirement from 0.30 to 0.32 w/ API update [#5457](https://github.com/apache/arrow-datafusion/pull/5457) (alamb) +- Allow setting config extensions for TaskContext [#5497](https://github.com/apache/arrow-datafusion/pull/5497) (mpurins-coralogix) +- Minor: Improve docs for UserDefinedLogicalNode `dyn_eq` and `dyn_hash` [#5515](https://github.com/apache/arrow-datafusion/pull/5515) (alamb) +- feat: interval add timestamp [#5491](https://github.com/apache/arrow-datafusion/pull/5491) (Weijun-H) +- Pass booleans by value instead of by reference [#5487](https://github.com/apache/arrow-datafusion/pull/5487) (maxburke) +- Minor: Move TableProviderFactories up out of `RuntimeEnv` and into `SessionState` [#5477](https://github.com/apache/arrow-datafusion/pull/5477) (alamb) +- feat: `ParquetExec` predicate preservation [#5495](https://github.com/apache/arrow-datafusion/pull/5495) (crepererum) +- feat: add optimization rules for bitwise operations [#5423](https://github.com/apache/arrow-datafusion/pull/5423) (izveigor) +- chore: Remove references from SessionState from physical_plan [#5455](https://github.com/apache/arrow-datafusion/pull/5455) (alamb) +- Implement `Debug` for `ExecutionProps` and `VarProvider` [#5489](https://github.com/apache/arrow-datafusion/pull/5489) (alamb) +- feat: Support bitwise operations for unsigned integer types [#5476](https://github.com/apache/arrow-datafusion/pull/5476) (izveigor) +- Apply workaround for #5444 to `DataFrame::describe` [#5468](https://github.com/apache/arrow-datafusion/pull/5468) (jiangzhx) +- feat: eliminate the duplicated sort keys in Order By clause [#5462](https://github.com/apache/arrow-datafusion/pull/5462) (jackwener) +- Propagate timezone to created arrays [#5481](https://github.com/apache/arrow-datafusion/pull/5481) (maxburke) +- refactor: make GeometricMean not to have update and merge [#5469](https://github.com/apache/arrow-datafusion/pull/5469) (Weijun-H) +- feat: add name() method to UserDefinedLogicalNode [#5450](https://github.com/apache/arrow-datafusion/pull/5450) (waynexia) +- Comment out description text in issue templates [#5482](https://github.com/apache/arrow-datafusion/pull/5482) (Jefffrey) +- feat: express unsigned literal in substrait [#5448](https://github.com/apache/arrow-datafusion/pull/5448) (waynexia) +- fix: build union schema with child has same column name but qualifier… [#5452](https://github.com/apache/arrow-datafusion/pull/5452) (yukkit) +- refactor: make sum_distinct not to have update and merge [#5474](https://github.com/apache/arrow-datafusion/pull/5474) (Weijun-H) +- `compute_decimal_op_dyn_scalar` should not cast lhs array to decimal array [#5465](https://github.com/apache/arrow-datafusion/pull/5465) (viirya) +- feat: `extensions_options` macro [#5442](https://github.com/apache/arrow-datafusion/pull/5442) (crepererum) +- Enable hash joins on FixedSizeBinary columns [#5461](https://github.com/apache/arrow-datafusion/pull/5461) (maxburke) +- Fix is_distinct from for float NaN values [#5446](https://github.com/apache/arrow-datafusion/pull/5446) (comphead) +- Implement/fix Eq and Hash for Expr and LogicalPlan [#5421](https://github.com/apache/arrow-datafusion/pull/5421) (mslapek) +- [feat]:fast check has column [#5328](https://github.com/apache/arrow-datafusion/pull/5328) (suxiaogang223) +- Parquet sorting benchmark [#5433](https://github.com/apache/arrow-datafusion/pull/5433) (jaylmiller) +- refactor count_distinct to not to have update and merge [#5408](https://github.com/apache/arrow-datafusion/pull/5408) (Weijun-H) +- build(deps): update zstd requirement from 0.11 to 0.12 [#5458](https://github.com/apache/arrow-datafusion/pull/5458) (alamb) +- Upgrade bytes to 1.4 [#5460](https://github.com/apache/arrow-datafusion/pull/5460) (viirya) +- add std,median result to describe method [#5445](https://github.com/apache/arrow-datafusion/pull/5445) (jiangzhx) +- minor: Port more window tests to sqlogictests [#5434](https://github.com/apache/arrow-datafusion/pull/5434) (alamb) +- Use compute_op_dyn_scalar for datatime [#5315](https://github.com/apache/arrow-datafusion/pull/5315) (viirya) +- add a unit test that cover cast bug. [#5443](https://github.com/apache/arrow-datafusion/pull/5443) (jackwener) +- create new `datafusion-execution` crate, start splitting code out [#5432](https://github.com/apache/arrow-datafusion/pull/5432) (alamb) +- minor: fix clippy in nightly. [#5440](https://github.com/apache/arrow-datafusion/pull/5440) (jackwener) +- Support for Sliding Windows Joins with Symmetric Hash Join (SHJ) [#5322](https://github.com/apache/arrow-datafusion/pull/5322) (metesynnada) +- refactor: ParquetExec logical expr. => phys. expr. [#5419](https://github.com/apache/arrow-datafusion/pull/5419) (crepererum) +- Update README.md fix [DataFusion] links [#5438](https://github.com/apache/arrow-datafusion/pull/5438) (jiangzhx) +- add mean result for describe method [#5435](https://github.com/apache/arrow-datafusion/pull/5435) (jiangzhx) +- add expr_fn::median [#5437](https://github.com/apache/arrow-datafusion/pull/5437) (jiangzhx) +- Bug/union wrong casting [#5342](https://github.com/apache/arrow-datafusion/pull/5342) (berkaysynnada) +- reimplement `push_down_projection` and `prune_column`. [#4465](https://github.com/apache/arrow-datafusion/pull/4465) (jackwener) +- Add `expr_fn::stddev` [#5409](https://github.com/apache/arrow-datafusion/pull/5409) (jiangzhx) +- fix nested loop join with literal join filter [#5431](https://github.com/apache/arrow-datafusion/pull/5431) (ygf11) +- add a describe method on DataFrame like Polars [#5226](https://github.com/apache/arrow-datafusion/pull/5226) (jiangzhx) +- Memory reservation & metrics for cross join [#5339](https://github.com/apache/arrow-datafusion/pull/5339) (korowa) +- Optimize count_distinct.size [#5377](https://github.com/apache/arrow-datafusion/pull/5377) (comphead) +- Fix filter pushdown for extension plans [#5425](https://github.com/apache/arrow-datafusion/pull/5425) (thinkharderdev) +- Also push down all filters in TableProvider [#5420](https://github.com/apache/arrow-datafusion/pull/5420) (avantgardnerio) +- Update arrow 34 [#5375](https://github.com/apache/arrow-datafusion/pull/5375) (tustvold) +- Parquet limit pushdown (#5404) [#5416](https://github.com/apache/arrow-datafusion/pull/5416) (tustvold) +- Move file format config.rs to live with the rest of the datasource code [#5406](https://github.com/apache/arrow-datafusion/pull/5406) (alamb) +- Support Zstd compressed files [#5397](https://github.com/apache/arrow-datafusion/pull/5397) (dennybritz) +- Add example of catalog API usage (#5291) [#5326](https://github.com/apache/arrow-datafusion/pull/5326) (jaylmiller) +- Add support for protobuf serialisation of Arrow Map type [#5359](https://github.com/apache/arrow-datafusion/pull/5359) (ahmedriza) +- minor: port window tests to slt (part 2) [#5399](https://github.com/apache/arrow-datafusion/pull/5399) (alamb) +- fix(docs): fix typos [#5403](https://github.com/apache/arrow-datafusion/pull/5403) (WenyXu) +- Try to push down full filter before break-up [#5367](https://github.com/apache/arrow-datafusion/pull/5367) (avantgardnerio) +- enhance: remove more projection. [#5402](https://github.com/apache/arrow-datafusion/pull/5402) (jackwener) +- refactor `push_down_filter` to fix dead-loop and use optimizer_recurse. [#5337](https://github.com/apache/arrow-datafusion/pull/5337) (jackwener) +- feat: eliminate unnecessary projection. [#5366](https://github.com/apache/arrow-datafusion/pull/5366) (jackwener) +- minor: add forgotten large_utf8 [#5393](https://github.com/apache/arrow-datafusion/pull/5393) (jackwener) +- Minor: add tests for subquery to join [#5363](https://github.com/apache/arrow-datafusion/pull/5363) (ygf11) +- bugfix: fix master `bors` problem. [#5395](https://github.com/apache/arrow-datafusion/pull/5395) (jackwener) +- Rule ReplaceDistinctWithAggregate [#5354](https://github.com/apache/arrow-datafusion/pull/5354) (mingmwang) +- chore: add known project ZincObserve [#5376](https://github.com/apache/arrow-datafusion/pull/5376) (hengfeiyang) +- refactor: parquet pruning simplifications [#5386](https://github.com/apache/arrow-datafusion/pull/5386) (crepererum) +- Minor: intersect expressions optimization [#5388](https://github.com/apache/arrow-datafusion/pull/5388) (izveigor) +- docs: clarify spark [#5391](https://github.com/apache/arrow-datafusion/pull/5391) (hyoklee) +- UDF zero params #5378 [#5380](https://github.com/apache/arrow-datafusion/pull/5380) (jaylmiller) +- Minor: added some tests for coercion type [#5389](https://github.com/apache/arrow-datafusion/pull/5389) (izveigor) +- minor: make table resolution an independent function ... [#5373](https://github.com/apache/arrow-datafusion/pull/5373) (MichaelScofield) +- minor: port predicates tests to sqllogictests [#5374](https://github.com/apache/arrow-datafusion/pull/5374) (jackwener) +- Bug fix: Window frame range value outside the type range [#5384](https://github.com/apache/arrow-datafusion/pull/5384) (mustafasrepo) +- Fixed small typos in files of the optimizer [#5356](https://github.com/apache/arrow-datafusion/pull/5356) (izveigor) +- fix: misc phys. expression display bugs [#5387](https://github.com/apache/arrow-datafusion/pull/5387) (crepererum) +- Prepare for 19.0.0 release [#5381](https://github.com/apache/arrow-datafusion/pull/5381) (andygrove) +- minor: disable tpcds-q41 due to not support decorrelate disjunction subquery [#5369](https://github.com/apache/arrow-datafusion/pull/5369) (jackwener) ## [19.0.0](https://github.com/apache/arrow-datafusion/tree/19.0.0) (2023-02-24)