Skip to content

Request DataShard compaction if scheme has been changed #11147

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
maximyurchuk merged 20 commits into from
Nov 12, 2024
Merged

Request DataShard compaction if scheme has been changed #11147

maximyurchuk merged 20 commits into from
Nov 12, 2024

Conversation

@kungasc
Copy link
Contributor

@kungasc kungasc commented Oct 31, 2024 •
edited
Loading

Changelog entry

...

Changelog category

  • Not for changelog (changelog entry is not required)

Additional information

#7718

@github-actions
Copy link

github-actions bot commented Oct 31, 2024 •
edited
Loading

⚪ 2024-10-31 11:33:36 UTC Pre-commit check linux-x86_64-release-asan for 9a2483f has started.
⚪ 2024-10-31 11:33:46 UTC Artifacts will be uploaded here
⚪ 2024-10-31 11:37:08 UTC ya make is running...
🔴 2024-10-31 13:20:21 UTC Too many tests failed, NOT going to retry
🟡 2024-10-31 13:22:30 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10715 9524 0 636 502 53

🟢 2024-10-31 13:23:31 UTC Build successful.
🟢 2024-10-31 13:24:07 UTC ydbd size 5.7 GiB changed* by +34.0 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 55a88ca merge: 9a2483f diff diff %
ydbd size 6 149 801 808 Bytes 6 149 836 632 Bytes +34.0 KiB +0.001%
ydbd stripped size 1 534 429 616 Bytes 1 534 437 808 Bytes +8.0 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Oct 31, 2024 •
edited
Loading

⚪ 2024-10-31 11:33:57 UTC Pre-commit check linux-x86_64-relwithdebinfo for 9a2483f has started.
⚪ 2024-10-31 11:34:09 UTC Artifacts will be uploaded here
⚪ 2024-10-31 11:37:40 UTC ya make is running...
🔴 2024-10-31 13:09:58 UTC Too many tests failed, NOT going to retry
🔴 2024-10-31 13:11:26 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
41744 34072 0 853 6735 84

🟢 2024-10-31 13:14:58 UTC Build successful.
🟢 2024-10-31 13:15:21 UTC ydbd size 2.8 GiB changed* by +14.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 55a88ca merge: 9a2483f diff diff %
ydbd size 3 039 299 072 Bytes 3 039 314 136 Bytes +14.7 KiB +0.000%
ydbd stripped size 481 329 016 Bytes 481 330 680 Bytes +1.6 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Oct 31, 2024 •
edited
Loading

⚪ 2024-10-31 16:16:10 UTC Pre-commit check linux-x86_64-release-asan for e1e08cd has started.
⚪ 2024-10-31 16:17:17 UTC Artifacts will be uploaded here
⚪ 2024-10-31 16:20:52 UTC ya make is running...
🔴 2024-10-31 16:54:11 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Oct 31, 2024 •
edited
Loading

⚪ 2024-10-31 16:17:53 UTC Pre-commit check linux-x86_64-relwithdebinfo for e1e08cd has started.
⚪ 2024-10-31 16:18:35 UTC Artifacts will be uploaded here
⚪ 2024-10-31 16:21:58 UTC ya make is running...
🔴 2024-10-31 16:48:44 UTC Build failed, see the logs. Also see fail summary

@kungasc kungasc linked an issue Oct 31, 2024 that may be closed by this pull request
Forcibly compact idle shards to build B-Tree index #7718
Closed
@github-actions
Copy link

github-actions bot commented Nov 1, 2024 •
edited
Loading

⚪ 2024-11-01 10:12:28 UTC Pre-commit check linux-x86_64-release-asan for 100e6c4 has started.
⚪ 2024-11-01 10:12:40 UTC Artifacts will be uploaded here
⚪ 2024-11-01 10:16:03 UTC ya make is running...
🔴 2024-11-01 10:44:03 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Nov 1, 2024 •
edited
Loading

⚪ 2024-11-01 10:14:43 UTC Pre-commit check linux-x86_64-relwithdebinfo for 100e6c4 has started.
⚪ 2024-11-01 10:14:54 UTC Artifacts will be uploaded here
⚪ 2024-11-01 10:18:20 UTC ya make is running...
🔴 2024-11-01 10:47:13 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Nov 1, 2024 •
edited
Loading

⚪ 2024-11-01 12:27:10 UTC Pre-commit check linux-x86_64-relwithdebinfo for 6683f7b has started.
⚪ 2024-11-01 12:27:21 UTC Artifacts will be uploaded here
⚪ 2024-11-01 12:30:53 UTC ya make is running...
🔴 2024-11-01 13:00:49 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Nov 1, 2024 •
edited
Loading

⚪ 2024-11-01 12:27:17 UTC Pre-commit check linux-x86_64-release-asan for 6683f7b has started.
⚪ 2024-11-01 12:27:28 UTC Artifacts will be uploaded here
⚪ 2024-11-01 12:31:01 UTC ya make is running...
⚫ 2024-11-01 13:04:24 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Nov 1, 2024 •
edited
Loading

⚪ 2024-11-01 13:08:07 UTC Pre-commit check linux-x86_64-relwithdebinfo for fa7c49c has started.
⚪ 2024-11-01 13:08:18 UTC Artifacts will be uploaded here
⚪ 2024-11-01 13:11:56 UTC ya make is running...
🔴 2024-11-01 13:43:35 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Nov 1, 2024 •
edited
Loading

⚪ 2024-11-01 13:08:46 UTC Pre-commit check linux-x86_64-release-asan for fa7c49c has started.
⚪ 2024-11-01 13:08:57 UTC Artifacts will be uploaded here
⚪ 2024-11-01 13:12:24 UTC ya make is running...
🔴 2024-11-01 13:38:20 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Nov 4, 2024 •
edited
Loading

⚪ 2024-11-04 10:35:40 UTC Pre-commit check linux-x86_64-release-asan for 6d215df has started.
⚪ 2024-11-04 10:36:17 UTC Artifacts will be uploaded here
⚫ 2024-11-04 10:37:19 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Nov 4, 2024 •
edited
Loading

⚪ 2024-11-04 10:36:46 UTC Pre-commit check linux-x86_64-relwithdebinfo for 6d215df has started.
⚪ 2024-11-04 10:36:57 UTC Artifacts will be uploaded here
⚫ 2024-11-04 10:37:19 UTC Check cancelled

@kungasc kungasc changed the title Report schema changes existence to Scheme Shard Request DataShard compaction if scheme has been changeed Nov 4, 2024
@github-actions
Copy link

github-actions bot commented Nov 4, 2024 •
edited
Loading

⚪ 2024-11-04 10:37:55 UTC Pre-commit check linux-x86_64-release-asan for d51be66 has started.
⚪ 2024-11-04 10:37:59 UTC Artifacts will be uploaded here
⚪ 2024-11-04 10:41:25 UTC ya make is running...
🔴 2024-11-04 11:13:04 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Nov 4, 2024 •
edited
Loading

⚪ 2024-11-04 10:40:57 UTC Pre-commit check linux-x86_64-relwithdebinfo for d51be66 has started.
⚪ 2024-11-04 10:41:10 UTC Artifacts will be uploaded here
⚪ 2024-11-04 10:44:46 UTC ya make is running...
🔴 2024-11-04 11:09:06 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Nov 4, 2024 •
edited
Loading

⚪ 2024-11-04 11:53:10 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4ceda63 has started.
⚪ 2024-11-04 11:53:21 UTC Artifacts will be uploaded here
⚫ 2024-11-04 11:55:19 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Nov 4, 2024 •
edited
Loading

⚪ 2024-11-04 11:53:19 UTC Pre-commit check linux-x86_64-release-asan for 4ceda63 has started.
⚪ 2024-11-04 11:53:31 UTC Artifacts will be uploaded here
⚫ 2024-11-04 11:55:19 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Nov 4, 2024 •
edited
Loading

⚪ 2024-11-04 11:57:30 UTC Pre-commit check linux-x86_64-relwithdebinfo for d4fb023 has started.
⚪ 2024-11-04 11:57:41 UTC Artifacts will be uploaded here
⚪ 2024-11-04 12:01:06 UTC ya make is running...
🔴 2024-11-04 12:22:47 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Nov 4, 2024 •
edited
Loading

⚪ 2024-11-04 11:57:39 UTC Pre-commit check linux-x86_64-release-asan for d4fb023 has started.
⚪ 2024-11-04 11:57:50 UTC Artifacts will be uploaded here
⚪ 2024-11-04 12:01:14 UTC ya make is running...
🔴 2024-11-04 12:25:41 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Nov 4, 2024 •
edited
Loading

⚪ 2024-11-04 13:40:30 UTC Pre-commit check linux-x86_64-release-asan for b5d65b5 has started.
⚪ 2024-11-04 13:40:41 UTC Artifacts will be uploaded here
⚪ 2024-11-04 13:44:11 UTC ya make is running...
⚫ 2024-11-04 14:01:14 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Nov 4, 2024 •
edited
Loading

⚪ 2024-11-04 13:40:45 UTC Pre-commit check linux-x86_64-relwithdebinfo for b5d65b5 has started.
⚪ 2024-11-04 13:40:55 UTC Artifacts will be uploaded here
⚪ 2024-11-04 13:44:19 UTC ya make is running...
🔴 2024-11-04 13:51:47 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Nov 4, 2024 •
edited
Loading

⚪ 2024-11-04 14:03:41 UTC Pre-commit check linux-x86_64-relwithdebinfo for 6277606 has started.
⚪ 2024-11-04 14:04:13 UTC Artifacts will be uploaded here
⚪ 2024-11-04 14:08:20 UTC ya make is running...
🔴 2024-11-04 14:18:57 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Nov 4, 2024 •
edited
Loading

⚪ 2024-11-04 14:05:51 UTC Pre-commit check linux-x86_64-release-asan for 6277606 has started.
⚪ 2024-11-04 14:06:02 UTC Artifacts will be uploaded here
⚪ 2024-11-04 14:09:36 UTC ya make is running...
🔴 2024-11-04 14:20:40 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Nov 4, 2024 •
edited
Loading

⚪ 2024-11-04 15:16:59 UTC Pre-commit check linux-x86_64-relwithdebinfo for 77eb97c has started.
⚪ 2024-11-04 15:17:11 UTC Artifacts will be uploaded here
⚪ 2024-11-04 15:20:44 UTC ya make is running...
🟡 2024-11-04 16:21:08 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
41715 36058 0 4 5558 95

⚪ 2024-11-04 16:24:50 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-11-04 16:42:44 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
98 (only retried tests) 6 0 0 0 92

🟢 2024-11-04 16:42:51 UTC Build successful.
🟢 2024-11-04 16:43:11 UTC ydbd size 2.8 GiB changed* by +26.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: f12f073 merge: 77eb97c diff diff %
ydbd size 3 038 072 456 Bytes 3 038 099 944 Bytes +26.8 KiB +0.001%
ydbd stripped size 481 429 176 Bytes 481 436 728 Bytes +7.4 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Nov 4, 2024 •
edited
Loading

⚪ 2024-11-04 15:19:45 UTC Pre-commit check linux-x86_64-release-asan for 77eb97c has started.
⚪ 2024-11-04 15:19:56 UTC Artifacts will be uploaded here
⚪ 2024-11-04 15:23:31 UTC ya make is running...
🟡 2024-11-04 16:43:00 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10685 10604 0 34 14 33

🟢 2024-11-04 16:43:54 UTC Build successful.
🟢 2024-11-04 16:44:27 UTC ydbd size 5.7 GiB changed* by +53.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: f12f073 merge: 77eb97c diff diff %
ydbd size 6 147 342 800 Bytes 6 147 397 464 Bytes +53.4 KiB +0.001%
ydbd stripped size 1 534 493 616 Bytes 1 534 510 960 Bytes +16.9 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@kungasc kungasc marked this pull request as ready for review November 5, 2024 08:18
@kungasc kungasc force-pushed the background-compaction-b-tree branch from 94c38c7 to 4ffdbbc Compare November 12, 2024 09:33
@github-actions
Copy link

github-actions bot commented Nov 12, 2024 •
edited
Loading

⚪ 2024-11-12 09:35:42 UTC Pre-commit check linux-x86_64-release-asan for 56808e6 has started.
⚪ 2024-11-12 09:35:54 UTC Artifacts will be uploaded here
⚪ 2024-11-12 09:39:17 UTC ya make is running...
🟡 2024-11-12 11:14:46 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10432 10339 0 37 8 48

🟢 2024-11-12 11:15:40 UTC Build successful.
🟢 2024-11-12 11:16:12 UTC ydbd size 5.7 GiB changed* by +79.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: e62a642 merge: 56808e6 diff diff %
ydbd size 6 172 635 312 Bytes 6 172 716 984 Bytes +79.8 KiB +0.001%
ydbd stripped size 1 538 451 376 Bytes 1 538 473 808 Bytes +21.9 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Nov 12, 2024 •
edited
Loading

⚪ 2024-11-12 09:36:10 UTC Pre-commit check linux-x86_64-relwithdebinfo for 56808e6 has started.
⚪ 2024-11-12 09:36:22 UTC Artifacts will be uploaded here
⚪ 2024-11-12 09:39:47 UTC ya make is running...
🟡 2024-11-12 11:07:24 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
41800 36117 0 12 5558 113

⚪ 2024-11-12 11:10:56 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-11-12 11:23:01 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
119 (only retried tests) 9 0 1 0 109

⚪ 2024-11-12 11:23:10 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-11-12 11:34:55 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
110 (only retried tests) 0 0 1 0 109

🟢 2024-11-12 11:35:08 UTC Build successful.
🟢 2024-11-12 11:35:29 UTC ydbd size 2.8 GiB changed* by +18.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: f1bf216 merge: 56808e6 diff diff %
ydbd size 3 050 203 200 Bytes 3 050 222 312 Bytes +18.7 KiB +0.001%
ydbd stripped size 482 471 224 Bytes 482 475 128 Bytes +3.8 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@kungasc kungasc requested a review from snaury November 12, 2024 11:25
@maximyurchuk maximyurchuk merged commit edeb17d into ydb-platform:main Nov 12, 2024
@kungasc kungasc changed the title Request DataShard compaction if scheme has been changeed Request DataShard compaction if scheme has been changed Nov 12, 2024
@GrigoriyPA GrigoriyPA mentioned this pull request Nov 12, 2024
Merged
@kungasc kungasc mentioned this pull request Nov 13, 2024
Merged
This was referenced Nov 13, 2024
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@snaury snaury snaury approved these changes

Assignees

No one assigned

Labels

not-for-changelog

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Forcibly compact idle shards to build B-Tree index

3 participants

@kungasc @snaury @maximyurchuk