WIP: df patched upgrade to 2024-03-05 #1
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… dictionaries (apache#9679) * Add test for multiple count distincts on a dictionary * Fix accumulator merge bug * Fix cleanup code
…r common subexpr elimination optimization (apache#9685) * test(9678): reproducer of short-circuiting causing expr elimination to error * fix(9678): populate visited stack for short-circuited expressions, during the common-expr elimination optimization * test(9678): reproducer for optimizer error (in common_subexpr_eliminate), as seen in other test case * chore: extract id_array into abstraction, to make it more clear the relationship between the two visitors * refactor: tweak the fix and make code more explicit (JumpMark, node_to_identifier) * fix: get the series_number and curr_id with the correct self.current_idx, before the various incr/decr * chore: remove unneeded conditional check (already done earlier), and add code comments * Refine documentation in common_subexpr_eliminate.rs * chore: cleanup -- fix 1 doc comment and consolidate common-expr-elimination test with other expr test --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
… not always stay in sync with the updated TreeNode traversal
…, while keeping the (stack-popped) symbol used for alias.
…expr_set, while keeping the (stack-popped) symbol used for alias." This reverts commit 049bf09.
…ing does not always stay in sync with the updated TreeNode traversal" This reverts commit d59a8de.
…re-find the correct expression during re-write. (apache#9871) * test(9870): reproducer of error with jumping traversal patterns in common-expr-elimination traversals * refactor: remove the IdArray ordered idx, since the idx ordering does not always stay in sync with the updated TreeNode traversal * refactor: use the only reproducible key (expr_identifer) for expr_set, while keeping the (stack-popped) symbol used for alias. * refactor: encapsulate most of the logic within ExprSet, and delineate the expr_identifier from the alias symbol * test(9870): demonstrate that the sqllogictests are now passing
… exist on main, but do exist at 2024-03-05
* fix: Remove supported coalesce types * Use comparison_coercion * Fix test * Fix * Add comment * More * fix
wiedld
changed the title
|
Already have advanced to a stable 2024-03-12 branch. Closing. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Below is edited, since a patch was merged into main
What's in this branch:
When testing against iox, we have been finding patches needed in DF. This is a branch for datafusion through EOD 2024-03-05, and then layering on patches needed.
Starting at datafusion main branch commit from March 5th 2024:
Then we added these commits:
COUNT(DISTINCT..)aggregates on dictionaries (Fix incorrect results with multipleCOUNT(DISTINCT..)aggregates on dictionaries apache/datafusion#9679), merged into datafusion on 2024-03-19.And a new patch, based upon a newly found bug:
10463#issuecomment-2024334683).
This new patch^^, merged into DF main on 2024-03-31, no longer had 2 methods which existed at 2024-03-05. Therefore, those two methods were patched (just for this 2024-03-05 branch):
The new DF patch (merged into main) also included a test using coalesce. This test relies upon a bug fix merged into main on March 7th (and not available on this 2024-03-05 branch). Added that patch too:
add the clippy build fix