feat(batch): introduce batch AsOf join #19790
Open
+230
−65
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chenzl25
reviewed
Dec 13, 2024
| stream_error: 'Invalid input syntax: AsOf join requires exactly 1 ineuquality condition' | ||
| - sql: CREATE TABLE t1(v1 varchar, v2 int, v3 int); CREATE TABLE t2(v1 varchar, v2 int, v3 int); SELECT t1.v1 t1_v1, t1.v2 t1_v2, t2.v1 t2_v1, t2.v2 t2_v2 FROM t1 ASOF JOIN t2 ON t1.v1 = t2.v1 || 'a' and t1.v2 > t2.v2; | ||
| batch_plan: |- | ||
| BatchExchange { order: [], dist: Single } | ||
| └─BatchGroupTopN { order: [t2.v2 DESC], limit: 1, offset: 0, group_key: [t1.v1, t1.v2] } |
There was a problem hiding this comment.
This plan seems incorrect because it will make the rows less then expected
Comment on lines
+17
to
+20
| query IIIIII | ||
| SELECT t1.v1 t1_v1, t1.v2 t1_v2, t1.v3 t1_v3, t2.v1 t2_v1, t2.v2 t2_v2, t2.v3 t2_v3 FROM t1 ASOF JOIN t2 ON t1.v1 = t2.v1 and t1.v2 < t2.v2 order by t1.v1; | ||
| ---- | ||
| 1 2 3 1 3 4 |
There was a problem hiding this comment.
Add one more row to the t1 and run this sql again.
insert into t1 values (1, 2, 9);The expected result should be (verified by duckdb)
┌───────┬───────┬───────┬───────┬───────┬───────┐
│ t1_v1 │ t1_v2 │ t1_v3 │ t2_v1 │ t2_v2 │ t2_v3 │
│ int32 │ int32 │ int32 │ int32 │ int32 │ int32 │
├───────┼───────┼───────┼───────┼───────┼───────┤
│ 1 │ 2 │ 3 │ 1 │ 3 │ 4 │
│ 1 │ 2 │ 9 │ 1 │ 3 │ 4 │
└───────┴───────┴───────┴───────┴───────┴───────┘
but current implementation will give us
t1_v1 | t1_v2 | t1_v3 | t2_v1 | t2_v2 | t2_v3
-------+-------+-------+-------+-------+-------
1 | 2 | 3 | 1 | 3 | 4
(1 row)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
plan the batch asof join as a hash join + a group top 1 + an optional projection. The group top 1 will be sorted by the right side asof column and group by the hash key. The optional projection is for pruning the additional asof column added for the GroupTopN.
Checklist
Documentation
Usage is the same same as #18683
Ties in the asof column in the right table.
The query result is non-deterministic when there are ties in the inequality condition column in the right table. ASOF JOIN aims to find and join the nearest record from the right table and only matches 1 row for each record from the left table. Therefore when there are ties in the right table, the right table row matched in the returned query result could be different in multiple runs.
Release note