Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: change stream logic #13808

Merged
merged 4 commits into from
Nov 30, 2023
Merged

chore: change stream logic #13808

merged 4 commits into from
Nov 30, 2023

Conversation

zhyass
Copy link
Member

@zhyass zhyass commented Nov 25, 2023

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

The stream filter:

Before:
not(is_not_null(_origin_block_id) and _origin_version<base_table_version)

Now:
not(is_not_null(_origin_version) and 
     (contains(_base_block_ids, _origin_block_id) or _origin_version<base_table_version))
mysql> create table t(a int) cluster by(a) change_tracking=true;
Query OK, 0 rows affected (0.04 sec)

mysql> alter table t set options(change_tracking=false);
Query OK, 0 rows affected (0.03 sec)

mysql> insert into t values(1),(2);
Query OK, 2 rows affected (0.03 sec)

mysql> alter table t set options(change_tracking=true);
Query OK, 0 rows affected (0.03 sec)

mysql> insert into t values(3),(4);
Query OK, 2 rows affected (0.04 sec)

mysql> select _origin_version,_origin_block_id,_origin_block_row_num,a from t order by a;
+-----------------+------------------+-----------------------+------+
| _origin_version | _origin_block_id | _origin_block_row_num | a    |
+-----------------+------------------+-----------------------+------+
|            NULL |             NULL |                  NULL |    1 |
|            NULL |             NULL |                  NULL |    2 |
|            NULL |             NULL |                  NULL |    3 |
|            NULL |             NULL |                  NULL |    4 |
+-----------------+------------------+-----------------------+------+
4 rows in set (0.03 sec)
Read 4 rows, 402.00 B in 0.016 sec., 254.5 rows/sec., 24.98 KiB/sec.

mysql> delete from t where a=3;
Query OK, 1 row affected (0.06 sec)

mysql> select _origin_version,_origin_block_id,_origin_block_row_num,a from t order by a;
+-----------------+-----------------------------------------+-----------------------+------+
| _origin_version | _origin_block_id                        | _origin_block_row_num | a    |
+-----------------+-----------------------------------------+-----------------------+------+
|            NULL |                                    NULL |                  NULL |    1 |
|            NULL |                                    NULL |                  NULL |    2 |
|            9114 | -70265286964003531484154567035708304538 |                     0 |    4 |
+-----------------+-----------------------------------------+-----------------------+------+
3 rows in set (0.03 sec)
Read 3 rows, 241.00 B in 0.009 sec., 346.51 rows/sec., 27.18 KiB/sec.

mysql> optimize table t compact;
Query OK, 3 rows affected (0.09 sec)

mysql> select _origin_version,_origin_block_id,_origin_block_row_num,a from t order by a;
+-----------------+-----------------------------------------+-----------------------+------+
| _origin_version | _origin_block_id                        | _origin_block_row_num | a    |
+-----------------+-----------------------------------------+-----------------------+------+
|            9119 |  99415263896031575503393984678632414870 |                     0 |    1 |
|            9119 |  99415263896031575503393984678632414870 |                     1 |    2 |
|            9114 | -70265286964003531484154567035708304538 |                     0 |    4 |
+-----------------+-----------------------------------------+-----------------------+------+
3 rows in set (0.03 sec)
Read 3 rows, 112.00 B in 0.008 sec., 375.51 rows/sec., 13.69 KiB/sec.

mysql> update t set a=3 where _origin_version=9114;
Query OK, 1 rows affected (0.07 sec)

mysql> select _origin_version,_origin_block_id,_origin_block_row_num,a from t order by a;
+-----------------+-----------------------------------------+-----------------------+------+
| _origin_version | _origin_block_id                        | _origin_block_row_num | a    |
+-----------------+-----------------------------------------+-----------------------+------+
|            9119 |  99415263896031575503393984678632414870 |                     0 |    1 |
|            9119 |  99415263896031575503393984678632414870 |                     1 |    2 |
|            9114 | -70265286964003531484154567035708304538 |                     0 |    3 |
+-----------------+-----------------------------------------+-----------------------+------+
3 rows in set (0.02 sec)
Read 3 rows, 112.00 B in 0.005 sec., 597.05 rows/sec., 21.77 KiB/sec.

mysql> insert into t values(5),(6);
Query OK, 2 rows affected (0.04 sec)

mysql> alter table t recluster;
Query OK, 5 rows affected (0.06 sec)

mysql> select _origin_version,_origin_block_id,_origin_block_row_num,a from t order by a;
+-----------------+-----------------------------------------+-----------------------+------+
| _origin_version | _origin_block_id                        | _origin_block_row_num | a    |
+-----------------+-----------------------------------------+-----------------------+------+
|            9119 |  99415263896031575503393984678632414870 |                     0 |    1 |
|            9119 |  99415263896031575503393984678632414870 |                     1 |    2 |
|            9114 | -70265286964003531484154567035708304538 |                     0 |    3 |
|            9131 |  24672370170860277704458166794042158056 |                     0 |    5 |
|            9131 |  24672370170860277704458166794042158056 |                     1 |    6 |
+-----------------+-----------------------------------------+-----------------------+------+
5 rows in set (0.04 sec)
Read 5 rows, 184.00 B in 0.008 sec., 589.88 rows/sec., 21.20 KiB/sec.
  • Closes #issue

This change is Reviewable

@github-actions github-actions bot added the pr-chore this PR only has small changes that no need to record, like coding styles. label Nov 25, 2023
@zhyass zhyass marked this pull request as draft November 25, 2023 18:01
@zhyass zhyass marked this pull request as ready for review November 27, 2023 03:02
@zhyass zhyass requested a review from dantengsky November 27, 2023 03:51
@zhyass zhyass marked this pull request as draft November 27, 2023 12:49
@zhyass zhyass changed the title chore: add stream_count in snapshot chore: change stream logic Nov 29, 2023
@zhyass zhyass force-pushed the feat_stream branch 2 times, most recently from 98fcae9 to 8634e72 Compare November 29, 2023 11:54
@zhyass zhyass marked this pull request as ready for review November 29, 2023 11:55
@zhyass zhyass mentioned this pull request Nov 29, 2023
7 tasks
@zhyass zhyass requested a review from sundy-li November 29, 2023 12:08
@zhyass zhyass force-pushed the feat_stream branch 2 times, most recently from 4148682 to 89407b3 Compare November 29, 2023 16:14
@BohuTANG BohuTANG added the ci-cloud Build docker image for cloud test label Nov 30, 2023
Copy link
Contributor

Docker Image for PR

  • tag: pr-13808-c05707a

note: this image tag is only available for internal use,
please check the internal doc for more details.

@BohuTANG
Copy link
Member

Copy link
Member

@BohuTANG BohuTANG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@BohuTANG BohuTANG merged commit 2f7e1be into databendlabs:main Nov 30, 2023
68 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-cloud Build docker image for cloud test pr-chore this PR only has small changes that no need to record, like coding styles.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants