Skip to content

add feature flag: enable olap tiering #8719

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
swalrus1 merged 9 commits into from
Sep 5, 2024
Merged

add feature flag: enable olap tiering #8719

swalrus1 merged 9 commits into from
Sep 5, 2024

Conversation

@swalrus1
Copy link
Collaborator

@swalrus1 swalrus1 commented Sep 4, 2024 •
edited
Loading

  • Add feature flag EnableOlapTiering. When it is disabled, the following actions are not allowed:
    • Creating Tiers
    • Creating Tiering rules
    • Applying Tiering to column tables

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 10:19:27 UTC Pre-commit check linux-x86_64-release-asan for 4df2c7b has started.
⚪ 2024-09-04 10:22:27 UTC ya make is running...
⚫ 2024-09-04 11:18:27 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 10:20:21 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4df2c7b has started.
⚪ 2024-09-04 10:23:32 UTC ya make is running...
⚫ 2024-09-04 11:18:46 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 10:21:41 UTC Pre-commit check linux-x86_64-release-clang14 for 4df2c7b has started.
⚪ 2024-09-04 10:24:56 UTC ya make is running...
🟢 2024-09-04 11:16:33 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 11:20:57 UTC Pre-commit check linux-x86_64-release-clang14 for db11e14 has started.
⚪ 2024-09-04 11:24:04 UTC ya make is running...
🟢 2024-09-04 11:40:20 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 11:21:15 UTC Pre-commit check linux-x86_64-release-asan for db11e14 has started.
⚪ 2024-09-04 11:24:17 UTC ya make is running...
⚫ 2024-09-04 11:45:10 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 11:22:13 UTC Pre-commit check linux-x86_64-relwithdebinfo for db11e14 has started.
⚪ 2024-09-04 11:25:18 UTC ya make is running...
⚫ 2024-09-04 11:45:25 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 11:47:44 UTC Pre-commit check linux-x86_64-release-asan for 013bb81 has started.
⚪ 2024-09-04 11:50:56 UTC ya make is running...
⚫ 2024-09-04 13:10:50 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 11:48:44 UTC Pre-commit check linux-x86_64-release-clang14 for 013bb81 has started.
⚪ 2024-09-04 11:51:45 UTC ya make is running...
🟢 2024-09-04 11:58:59 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 11:51:03 UTC Pre-commit check linux-x86_64-relwithdebinfo for 013bb81 has started.
⚪ 2024-09-04 11:54:07 UTC ya make is running...
🟡 2024-09-04 13:02:54 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40762 35320 0 11 5410 21

⚪ 2024-09-04 13:06:35 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-09-04 13:11:13 UTC ydbd size 8.3 GiB changed* by +183.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 376e237 merge: 013bb81 diff diff %
ydbd size 8 921 490 704 Bytes 8 921 678 904 Bytes +183.8 KiB +0.002%
ydbd stripped size 481 969 608 Bytes 481 971 624 Bytes +2.0 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation
⚫ 2024-09-04 13:11:15 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 13:15:15 UTC Pre-commit check linux-x86_64-release-clang14 for 1e02a53 has started.
⚪ 2024-09-04 13:18:50 UTC ya make is running...
🟢 2024-09-04 13:29:00 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 13:21:05 UTC Pre-commit check linux-x86_64-release-asan for 1e02a53 has started.
⚪ 2024-09-04 13:24:09 UTC ya make is running...
🔴 2024-09-04 14:58:51 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11509 11444 0 13 25 27

🟢 2024-09-04 15:00:20 UTC ydbd size 5.5 GiB changed* by +95.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: dd61f85 merge: 1e02a53 diff diff %
ydbd size 5 957 674 128 Bytes 5 957 771 816 Bytes +95.4 KiB +0.002%
ydbd stripped size 1 490 132 880 Bytes 1 490 143 344 Bytes +10.2 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation
⚫ 2024-09-04 15:00:22 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 13:21:47 UTC Pre-commit check linux-x86_64-relwithdebinfo for 1e02a53 has started.
⚪ 2024-09-04 13:26:07 UTC ya make is running...
🟡 2024-09-04 14:38:38 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40761 35278 0 11 5452 20

⚪ 2024-09-04 14:42:17 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-09-04 14:52:20 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
89 (only retried tests) 72 0 1 2 14

⚪ 2024-09-04 14:52:28 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-09-04 14:59:19 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
21 (only retried tests) 6 0 1 0 14

🟢 2024-09-04 14:59:26 UTC Build successful.
🟡 2024-09-04 15:00:06 UTC ydbd size 8.3 GiB changed* by +184.7 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: dd61f85 merge: 1e02a53 diff diff %
ydbd size 8 922 565 400 Bytes 8 922 754 496 Bytes +184.7 KiB +0.002%
ydbd stripped size 481 974 600 Bytes 481 976 488 Bytes +1.8 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation
⚫ 2024-09-04 15:00:09 UTC Check cancelled

@swalrus1 swalrus1 force-pushed the enable-tiering-feature-flag branch from ebca912 to 4843018 Compare September 4, 2024 14:59
@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 15:04:47 UTC Pre-commit check linux-x86_64-release-clang14 for 6f98320 has started.
⚪ 2024-09-04 15:07:53 UTC ya make is running...
🟢 2024-09-04 15:54:56 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 15:16:41 UTC Pre-commit check linux-x86_64-relwithdebinfo for 6f98320 has started.
⚪ 2024-09-04 15:20:58 UTC ya make is running...
⚫ 2024-09-04 16:02:36 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 15:17:09 UTC Pre-commit check linux-x86_64-release-asan for 6f98320 has started.
⚪ 2024-09-04 15:21:20 UTC ya make is running...
⚫ 2024-09-04 16:02:35 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 16:14:34 UTC Pre-commit check linux-x86_64-release-asan for c398ea6 has started.
⚪ 2024-09-04 16:18:43 UTC ya make is running...
🔴 2024-09-04 18:24:20 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11518 11449 0 22 24 23

🟢 2024-09-04 18:25:20 UTC Build successful.
🟢 2024-09-04 18:25:49 UTC ydbd size 5.6 GiB changed* by +93.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: f8bebff merge: c398ea6 diff diff %
ydbd size 6 019 217 752 Bytes 6 019 313 888 Bytes +93.9 KiB +0.002%
ydbd stripped size 1 507 000 912 Bytes 1 507 010 224 Bytes +9.1 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 16:27:20 UTC Pre-commit check linux-x86_64-relwithdebinfo for c398ea6 has started.
⚪ 2024-09-04 16:31:12 UTC ya make is running...
🟡 2024-09-04 18:14:07 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40774 35344 0 5 5403 22

⚪ 2024-09-04 18:17:58 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-09-04 18:25:17 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
35 (only retried tests) 20 0 0 0 15

🟢 2024-09-04 18:25:24 UTC Build successful.
🟡 2024-09-04 18:26:00 UTC ydbd size 8.4 GiB changed* by +185.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 93bd594 merge: c398ea6 diff diff %
ydbd size 9 000 764 872 Bytes 9 000 955 232 Bytes +185.9 KiB +0.002%
ydbd stripped size 486 645 096 Bytes 486 650 664 Bytes +5.4 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 4, 2024 •
edited
Loading

⚪ 2024-09-04 16:28:00 UTC Pre-commit check linux-x86_64-release-clang14 for c398ea6 has started.
⚪ 2024-09-04 16:31:22 UTC ya make is running...
🟢 2024-09-04 17:16:44 UTC Build successful.

@swalrus1 swalrus1 force-pushed the enable-tiering-feature-flag branch from 7db45c1 to 23065e7 Compare September 5, 2024 07:37
@github-actions
Copy link

github-actions bot commented Sep 5, 2024 •
edited
Loading

⚪ 2024-09-05 07:38:42 UTC Pre-commit check linux-x86_64-release-clang14 for 8f996dc has started.
⚪ 2024-09-05 07:41:56 UTC ya make is running...
🟢 2024-09-05 08:34:13 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Sep 5, 2024 •
edited
Loading

⚪ 2024-09-05 07:39:12 UTC Pre-commit check linux-x86_64-release-asan for 8f996dc has started.
⚪ 2024-09-05 07:42:32 UTC ya make is running...
🔴 2024-09-05 10:15:23 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11535 11404 0 51 43 37

🟢 2024-09-05 10:16:22 UTC Build successful.
🟢 2024-09-05 10:17:02 UTC ydbd size 5.6 GiB changed* by +27.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 732bf12 merge: 8f996dc diff diff %
ydbd size 6 019 637 112 Bytes 6 019 664 864 Bytes +27.1 KiB +0.000%
ydbd stripped size 1 507 074 288 Bytes 1 507 083 888 Bytes +9.4 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 5, 2024 •
edited
Loading

⚪ 2024-09-05 07:40:47 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8f996dc has started.
⚪ 2024-09-05 07:43:57 UTC ya make is running...
🟡 2024-09-05 09:34:58 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40786 35328 0 12 5422 24

⚪ 2024-09-05 09:38:31 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-09-05 09:46:11 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
52 (only retried tests) 32 0 1 0 19

⚪ 2024-09-05 09:46:18 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-09-05 09:53:25 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
26 (only retried tests) 6 0 1 0 19

🟢 2024-09-05 09:53:32 UTC Build successful.
🟢 2024-09-05 09:54:09 UTC ydbd size 8.4 GiB changed* by +21.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 732bf12 merge: 8f996dc diff diff %
ydbd size 9 001 104 776 Bytes 9 001 126 784 Bytes +21.5 KiB +0.000%
ydbd stripped size 486 652 968 Bytes 486 658 664 Bytes +5.6 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@swalrus1 swalrus1 marked this pull request as ready for review September 5, 2024 08:01
@swalrus1 swalrus1 enabled auto-merge (squash) September 5, 2024 08:01
@swalrus1 swalrus1 force-pushed the enable-tiering-feature-flag branch from 23065e7 to 43542d1 Compare September 5, 2024 10:44
@github-actions
Copy link

github-actions bot commented Sep 5, 2024 •
edited
Loading

⚪ 2024-09-05 11:02:19 UTC Pre-commit check linux-x86_64-release-asan for 2ea2205 has started.
⚪ 2024-09-05 11:05:28 UTC ya make is running...
🔴 2024-09-05 12:41:46 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11506 11424 0 21 31 30

🟢 2024-09-05 12:42:46 UTC Build successful.
🟢 2024-09-05 12:43:16 UTC ydbd size 5.6 GiB changed* by +26.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: d25dbce merge: 2ea2205 diff diff %
ydbd size 6 019 637 416 Bytes 6 019 664 816 Bytes +26.8 KiB +0.000%
ydbd stripped size 1 507 074 480 Bytes 1 507 083 952 Bytes +9.2 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 5, 2024 •
edited
Loading

⚪ 2024-09-05 11:09:06 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2ea2205 has started.
⚪ 2024-09-05 11:13:38 UTC ya make is running...
🟡 2024-09-05 12:30:11 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40785 35340 0 7 5415 23

⚪ 2024-09-05 12:34:07 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-09-05 12:43:02 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
41 (only retried tests) 23 0 0 0 18

🟢 2024-09-05 12:43:10 UTC Build successful.
🟢 2024-09-05 12:43:47 UTC ydbd size 8.4 GiB changed* by +17.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: fe3a015 merge: 2ea2205 diff diff %
ydbd size 9 001 108 512 Bytes 9 001 126 488 Bytes +17.6 KiB +0.000%
ydbd stripped size 486 657 064 Bytes 486 658 728 Bytes +1.6 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 5, 2024 •
edited
Loading

⚪ 2024-09-05 11:10:23 UTC Pre-commit check linux-x86_64-release-clang14 for 2ea2205 has started.
⚪ 2024-09-05 11:13:43 UTC ya make is running...
🟢 2024-09-05 11:19:09 UTC Build successful.

@swalrus1 swalrus1 merged commit dc3cf10 into ydb-platform:main Sep 5, 2024
@swalrus1 swalrus1 deleted the enable-tiering-feature-flag branch September 5, 2024 12:46
ijon
ijon reviewed Sep 6, 2024
View reviewed changes
ydb/core/tx/schemeshard/olap/operations/alter_table.cpp

const bool hasTiering = Transaction.HasAlterColumnTable() && Transaction.GetAlterColumnTable().HasAlterTtlSettings() &&
Transaction.GetAlterColumnTable().GetAlterTtlSettings().HasUseTiering();
if (hasTiering && HasAppData() && !AppDataVerified().FeatureFlags.GetEnableTieringInColumnShard()) {
Copy link
Collaborator

@ijon ijon Sep 6, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HasAppData() && !AppDataVerified() -- лишняя защита, здесь мы заведомо находимся в контексте акторной системы.

В других подобных местах так не делается.

ydb/core/tx/schemeshard/olap/operations/alter_table.cpp

const bool hasTiering = Transaction.HasAlterColumnTable() && Transaction.GetAlterColumnTable().HasAlterTtlSettings() &&
Transaction.GetAlterColumnTable().GetAlterTtlSettings().HasUseTiering();
if (hasTiering && HasAppData() && !AppDataVerified().FeatureFlags.GetEnableTieringInColumnShard()) {
Copy link
Collaborator

@ijon ijon Sep 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Таким образом использующийся флаг будет нельзя менять в runtime -- только через рестарт нод базы

@niksaveliev niksaveliev mentioned this pull request Sep 9, 2024
Merged
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 10, 2024
@swalrus1 @zverevgeny
add feature flag: enable olap tiering (ydb-platform#8719)
2ca6355 
Conflicts:
	ydb/core/protos/feature_flags.proto
	ydb/core/testlib/basics/feature_flags.h
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 11, 2024
@swalrus1 @zverevgeny
add feature flag: enable olap tiering (ydb-platform#8719)
810473f 
Conflicts:
	ydb/core/protos/feature_flags.proto
	ydb/core/testlib/basics/feature_flags.h
@niksaveliev niksaveliev mentioned this pull request Sep 12, 2024
Merged
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 13, 2024
@swalrus1 @zverevgeny
add feature flag: enable olap tiering (ydb-platform#8719)
7367429 
Conflicts:
	ydb/core/protos/feature_flags.proto
zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Sep 15, 2024
@swalrus1 @zverevgeny
add feature flag: enable olap tiering (ydb-platform#8719)
75674bf 
Conflicts:
	ydb/core/protos/feature_flags.proto
@zverevgeny zverevgeny mentioned this pull request Sep 16, 2024
Closed
@StekPerepolnen StekPerepolnen mentioned this pull request Sep 17, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ijon ijon ijon left review comments

@ivanmorozov333 ivanmorozov333 ivanmorozov333 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@swalrus1 @ijon @ivanmorozov333