Skip to content

Conversation

@UgnineSirdis
Copy link
Collaborator

Changelog entry

...

Changelog category

  • Not for changelog (changelog entry is not required)

Description for reviewers

In this PR I add a special stage to index build process: validation of built index. This is used only as a unique index build stage.

Also here is the fix of the bug: in KQP index build case we don't fail query if index build process fails.

@github-actions
Copy link

github-actions bot commented Jun 24, 2025

2025-06-24 17:24:09 UTC Pre-commit check linux-x86_64-release-asan for 4ad4b45 has started.
2025-06-24 17:24:21 UTC Artifacts will be uploaded here
2025-06-24 17:28:10 UTC ya make is running...
🟡 2025-06-24 19:59:07 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16046 15680 0 124 209 33

🟢 2025-06-24 20:00:33 UTC Build successful.
🟡 2025-06-24 20:01:02 UTC ydbd size 3.9 GiB changed* by +725.1 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 04c388e merge: 4ad4b45 diff diff %
ydbd size 4 191 869 120 Bytes 4 192 611 576 Bytes +725.1 KiB +0.018%
ydbd stripped size 1 453 112 632 Bytes 1 453 360 536 Bytes +242.1 KiB +0.017%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Jun 24, 2025

2025-06-24 17:24:24 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4ad4b45 has started.
2025-06-24 17:24:37 UTC Artifacts will be uploaded here
2025-06-24 17:28:16 UTC ya make is running...
🟡 2025-06-24 19:20:13 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
38376 35659 0 5 2672 40

2025-06-24 19:23:29 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-06-24 19:36:06 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
767 (only retried tests) 705 0 2 33 27

2025-06-24 19:36:17 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-06-24 19:47:19 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
468 (only retried tests) 436 0 1 5 26

🟢 2025-06-24 19:47:27 UTC Build successful.
🟡 2025-06-24 19:47:48 UTC ydbd size 2.2 GiB changed* by +437.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 04c388e merge: 4ad4b45 diff diff %
ydbd size 2 382 766 744 Bytes 2 383 214 896 Bytes +437.6 KiB +0.019%
ydbd stripped size 498 847 400 Bytes 498 928 680 Bytes +79.4 KiB +0.016%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

🟢 2025-06-24 17:25:30 UTC The validation of the Pull Request description is successful.

@github-actions
Copy link

github-actions bot commented Jun 24, 2025

2025-06-25 10:21:44 UTC Pre-commit check linux-x86_64-relwithdebinfo for 1e518d8 has started.
2025-06-25 10:21:56 UTC Artifacts will be uploaded here
2025-06-25 10:25:39 UTC ya make is running...
🟡 2025-06-25 11:37:41 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
38376 35647 0 8 2672 49

2025-06-25 11:41:04 UTC ya make is running... (failed tests rerun, try 2)

@github-actions
Copy link

github-actions bot commented Jun 24, 2025

2025-06-24 20:09:02 UTC Pre-commit check linux-x86_64-release-asan for 1e518d8 has started.
2025-06-24 20:09:14 UTC Artifacts will be uploaded here
2025-06-24 20:13:04 UTC ya make is running...
🟡 2025-06-24 22:38:19 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16046 15696 0 124 193 33

🟢 2025-06-24 22:39:41 UTC Build successful.
🟡 2025-06-24 22:40:08 UTC ydbd size 3.9 GiB changed* by +725.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: a6a72eb merge: 1e518d8 diff diff %
ydbd size 4 191 872 696 Bytes 4 192 615 088 Bytes +725.0 KiB +0.018%
ydbd stripped size 1 453 114 104 Bytes 1 453 361 944 Bytes +242.0 KiB +0.017%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@UgnineSirdis UgnineSirdis force-pushed the unique-index-validation-stage branch from b2e1d86 to ce1b363 Compare June 25, 2025 21:19
@github-actions
Copy link

github-actions bot commented Jun 25, 2025

2025-06-25 21:21:00 UTC Pre-commit check linux-x86_64-relwithdebinfo for cd61a92 has started.
2025-06-25 21:21:15 UTC Artifacts will be uploaded here
2025-06-25 21:25:01 UTC ya make is running...
🟡 2025-06-25 23:14:23 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
38398 35675 0 9 2675 39

2025-06-25 23:17:58 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-06-25 23:30:09 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
722 (only retried tests) 663 0 0 31 28

🟢 2025-06-25 23:30:19 UTC Build successful.
🟡 2025-06-25 23:30:39 UTC ydbd size 2.2 GiB changed* by +441.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 63058f0 merge: cd61a92 diff diff %
ydbd size 2 383 108 552 Bytes 2 383 560 664 Bytes +441.5 KiB +0.019%
ydbd stripped size 498 964 424 Bytes 499 049 672 Bytes +83.2 KiB +0.017%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Jun 25, 2025

2025-06-25 21:21:49 UTC Pre-commit check linux-x86_64-release-asan for cd61a92 has started.
2025-06-25 21:22:09 UTC Artifacts will be uploaded here
2025-06-25 21:26:00 UTC ya make is running...
🟡 2025-06-25 23:56:06 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16067 15643 0 120 271 33

🟢 2025-06-25 23:57:32 UTC Build successful.
🟡 2025-06-25 23:57:59 UTC ydbd size 3.9 GiB changed* by +721.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 63058f0 merge: cd61a92 diff diff %
ydbd size 4 192 746 592 Bytes 4 193 484 904 Bytes +721.0 KiB +0.018%
ydbd stripped size 1 453 455 672 Bytes 1 453 699 416 Bytes +238.0 KiB +0.017%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

snaury
snaury previously approved these changes Jun 26, 2025
Copy link
Member

@snaury snaury left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

По даташардам от меня окей (изменение на вид минимально инвазивное). Однако учитывайте, что дизайн уникальных индексов, которые не уникальные на уровне реализации в даташарде / локальной базе может быть ошибкой с точки зрения эффективности. Дело в том, что тогда любой point read в таком индексе на самом деле выливается в range read, который дороже в реализации. Но кажется я это уже говорил на каком-то из прошлых обсуждений.

@CyberROFL CyberROFL requested review from kungasc and vitalif June 26, 2025 11:22
@kungasc kungasc self-assigned this Jun 26, 2025
Copy link
Collaborator

@vitalif vitalif left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ну в общем я за то, чтобы ПР разделить, а статусы вторые убрать :-)

@github-actions
Copy link

github-actions bot commented Aug 10, 2025

2025-08-10 20:55:06 UTC Pre-commit check linux-x86_64-relwithdebinfo for e6c20eb has started.
2025-08-10 20:55:20 UTC Artifacts will be uploaded here
2025-08-10 20:59:15 UTC ya make is running...
🟡 2025-08-10 22:21:10 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39417 36617 0 3 2757 40

2025-08-10 22:24:40 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-08-10 22:37:57 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
844 (only retried tests) 767 0 0 53 24

🟢 2025-08-10 22:38:08 UTC Build successful.
🟡 2025-08-10 22:38:24 UTC ydbd size 2.2 GiB changed* by +126.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 29e0b20 merge: e6c20eb diff diff %
ydbd size 2 415 122 448 Bytes 2 415 251 920 Bytes +126.4 KiB +0.005%
ydbd stripped size 505 007 144 Bytes 505 035 336 Bytes +27.5 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Aug 10, 2025

2025-08-10 16:56:23 UTC Pre-commit check linux-x86_64-release-asan for e6c20eb has started.
2025-08-10 16:56:38 UTC Artifacts will be uploaded here
2025-08-10 17:00:30 UTC ya make is running...
🟡 2025-08-10 19:33:55 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16809 16374 0 103 306 26

🟢 2025-08-10 19:35:21 UTC Build successful.
🟡 2025-08-10 19:35:50 UTC ydbd size 4.0 GiB changed* by +223.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 29e0b20 merge: e6c20eb diff diff %
ydbd size 4 249 000 040 Bytes 4 249 229 344 Bytes +223.9 KiB +0.005%
ydbd stripped size 1 470 282 968 Bytes 1 470 365 496 Bytes +80.6 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Aug 19, 2025

2025-08-19 18:34:45 UTC Pre-commit check linux-x86_64-release-asan for 82d6a03 has started.
2025-08-19 18:34:58 UTC Artifacts will be uploaded here
2025-08-19 18:38:43 UTC ya make is running...
🟡 2025-08-19 21:11:18 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16983 16524 0 131 298 30

🟢 2025-08-19 21:12:47 UTC Build successful.
🟡 2025-08-19 21:13:15 UTC ydbd size 4.0 GiB changed* by +232.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 1b5a977 merge: 82d6a03 diff diff %
ydbd size 4 269 807 456 Bytes 4 270 045 056 Bytes +232.0 KiB +0.006%
ydbd stripped size 1 481 692 824 Bytes 1 481 775 256 Bytes +80.5 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Aug 21, 2025

2025-08-21 11:13:35 UTC Pre-commit check linux-x86_64-relwithdebinfo for 10b6182 has started.
2025-08-21 11:13:48 UTC Artifacts will be uploaded here
2025-08-21 11:17:45 UTC ya make is running...
2025-08-21 11:41:25 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Aug 21, 2025

2025-08-21 11:13:53 UTC Pre-commit check linux-x86_64-release-asan for 10b6182 has started.
2025-08-21 11:14:24 UTC Artifacts will be uploaded here
2025-08-21 11:18:52 UTC ya make is running...
2025-08-21 11:41:25 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Aug 21, 2025

2025-08-21 11:42:51 UTC Pre-commit check linux-x86_64-relwithdebinfo for 41dd481 has started.
2025-08-21 11:43:06 UTC Artifacts will be uploaded here
2025-08-21 11:47:03 UTC ya make is running...
🟡 2025-08-21 13:24:24 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39685 36928 0 1 2714 42

2025-08-21 13:27:46 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-08-21 13:41:30 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
501 (only retried tests) 474 0 0 1 26

🟢 2025-08-21 13:41:39 UTC Build successful.
🟡 2025-08-21 13:42:02 UTC ydbd size 2.3 GiB changed* by +146.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: d9db162 merge: 41dd481 diff diff %
ydbd size 2 429 138 832 Bytes 2 429 288 944 Bytes +146.6 KiB +0.006%
ydbd stripped size 509 252 392 Bytes 509 286 568 Bytes +33.4 KiB +0.007%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Y_ENSURE(found);
}

// Arrange shards in ascending order by shard index
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

а чего PerformCrossShardUniqIndexValidation сам внутри не посортирует как ему надо

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

и это не by shard index, а по ключу видимо

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

и что-то не пойму где оно сортируется вообще сейчас

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Я спрашивал у Ильназа и он сказал мне, что нужный мне порядок (по ключу) содержится в table->GetPartitions(). Сортировать отдельно смысла нет. Написал коммент, что "in ascending order by shard index"

@github-actions
Copy link

github-actions bot commented Aug 21, 2025

2025-08-21 11:44:24 UTC Pre-commit check linux-x86_64-release-asan for 41dd481 has started.
2025-08-21 11:44:43 UTC Artifacts will be uploaded here
2025-08-21 11:49:28 UTC ya make is running...
🟡 2025-08-21 14:13:20 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
17018 16564 0 131 291 32

🟢 2025-08-21 14:14:48 UTC Build successful.
🟡 2025-08-21 14:15:19 UTC ydbd size 4.0 GiB changed* by +235.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: d9db162 merge: 41dd481 diff diff %
ydbd size 4 268 775 568 Bytes 4 269 016 568 Bytes +235.4 KiB +0.006%
ydbd stripped size 1 481 373 272 Bytes 1 481 456 792 Bytes +81.6 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@UgnineSirdis UgnineSirdis dismissed vitalif’s stale review August 21, 2025 12:38

I refactored according to the request

@UgnineSirdis UgnineSirdis enabled auto-merge (squash) August 21, 2025 12:45
@UgnineSirdis UgnineSirdis merged commit 6771cfb into ydb-platform:main Aug 21, 2025
17 checks passed
UgnineSirdis added a commit to UgnineSirdis/ydb that referenced this pull request Aug 22, 2025
@UgnineSirdis UgnineSirdis deleted the unique-index-validation-stage branch August 26, 2025 09:42
UgnineSirdis added a commit that referenced this pull request Aug 26, 2025
SloNN pushed a commit to SloNN/ydb that referenced this pull request Aug 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants