Refactor FillVectorIndex state machine (#18579) #19128

vitalif · 2025-06-02T08:02:11Z

Changelog category

Not for changelog (changelog entry is not required)

github-actions · 2025-06-02T08:05:03Z

🟢 2025-06-02 08:05:02 UTC The validation of the Pull Request description is successful.

github-actions · 2025-06-02T08:05:55Z

⚪ 2025-06-02 08:05:55 UTC Pre-commit check linux-x86_64-release-asan for 538586f has started.
⚪ 2025-06-02 08:06:06 UTC Artifacts will be uploaded here
⚪ 2025-06-02 08:09:09 UTC ya make is running...
🟡 2025-06-02 09:52:17 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13794	13544	0	142	80	28

⚪ 2025-06-02 09:53:31 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-06-02 10:32:01 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
2098 (only retried tests)	2006	0	64	6	22

⚪ 2025-06-02 10:32:21 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-06-02 11:11:32 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
1801 (only retried tests)	1705	0	68	8	20

🟢 2025-06-02 11:11:49 UTC Build successful.
🟢 2025-06-02 11:12:25 UTC ydbd size 3.9 GiB changed* by +5.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `d70a7cc`	merge: `538586f`	diff	diff %
ydbd size	4 157 729 224 Bytes	4 157 735 216 Bytes	+5.9 KiB	+0.000%
ydbd stripped size	1 442 893 112 Bytes	1 442 894 072 Bytes	+960 Bytes	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-06-02T08:06:01Z

⚪ 2025-06-02 08:06:00 UTC Pre-commit check linux-x86_64-relwithdebinfo for 538586f has started.
⚪ 2025-06-02 08:06:12 UTC Artifacts will be uploaded here
⚪ 2025-06-02 08:09:18 UTC ya make is running...
🟡 2025-06-02 09:27:03 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
21774	20334	0	51	1349	40

⚪ 2025-06-02 09:28:55 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-06-02 09:45:19 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
874 (only retried tests)	839	0	5	0	30

⚪ 2025-06-02 09:45:31 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-06-02 10:00:17 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
793 (only retried tests)	757	0	4	0	32

🟢 2025-06-02 10:00:30 UTC Build successful.
🟢 2025-06-02 10:00:53 UTC ydbd size 2.2 GiB changed* by +4.0 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `d70a7cc`	merge: `538586f`	diff	diff %
ydbd size	2 364 055 928 Bytes	2 364 060 056 Bytes	+4.0 KiB	+0.000%
ydbd stripped size	496 255 400 Bytes	496 256 040 Bytes	+640 Bytes	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-06-02T13:32:36Z

⚪ 2025-06-02 13:32:36 UTC Pre-commit check linux-x86_64-relwithdebinfo for dd25fa1 has started.
⚪ 2025-06-02 13:32:47 UTC Artifacts will be uploaded here
⚪ 2025-06-02 13:35:56 UTC ya make is running...
🟡 2025-06-02 14:47:54 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
21785	20410	0	5	1338	32

⚪ 2025-06-02 14:49:43 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-06-02 15:01:56 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
479 (only retried tests)	455	0	0	0	24

🟢 2025-06-02 15:02:04 UTC Build successful.
🟢 2025-06-02 15:02:22 UTC ydbd size 2.2 GiB changed* by +4.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `e1df8e6`	merge: `dd25fa1`	diff	diff %
ydbd size	2 364 081 368 Bytes	2 364 085 640 Bytes	+4.2 KiB	+0.000%
ydbd stripped size	496 261 064 Bytes	496 261 768 Bytes	+704 Bytes	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-06-02T13:33:12Z

⚪ 2025-06-02 13:33:11 UTC Pre-commit check linux-x86_64-release-asan for dd25fa1 has started.
⚪ 2025-06-02 13:33:15 UTC Artifacts will be uploaded here
⚪ 2025-06-02 13:36:10 UTC ya make is running...
🟡 2025-06-02 15:12:32 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13801	13575	0	117	88	21

⚪ 2025-06-02 15:13:45 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-06-02 15:43:44 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
1410 (only retried tests)	1329	0	48	14	19

⚪ 2025-06-02 15:44:00 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-06-02 16:14:19 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
1300 (only retried tests)	1220	0	47	11	22

🟢 2025-06-02 16:14:34 UTC Build successful.
🟢 2025-06-02 16:15:04 UTC ydbd size 3.9 GiB changed* by +10.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `e1df8e6`	merge: `dd25fa1`	diff	diff %
ydbd size	4 157 789 904 Bytes	4 157 800 312 Bytes	+10.2 KiB	+0.000%
ydbd stripped size	1 442 911 384 Bytes	1 442 916 568 Bytes	+5.1 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-06-02T18:14:43Z

⚪ 2025-06-02 18:14:42 UTC Pre-commit check linux-x86_64-release-asan for 87cf12c has started.
⚪ 2025-06-02 18:14:53 UTC Artifacts will be uploaded here
⚪ 2025-06-02 18:17:57 UTC ya make is running...
🟡 2025-06-02 20:00:07 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13796	13607	0	87	76	26

⚪ 2025-06-02 20:01:18 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-06-02 20:35:23 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
1707 (only retried tests)	1628	0	52	5	22

⚪ 2025-06-02 20:35:41 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-06-02 21:05:16 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
1352 (only retried tests)	1278	0	46	4	24

🟢 2025-06-02 21:05:59 UTC Build successful.
🟢 2025-06-02 21:06:33 UTC ydbd size 3.9 GiB changed* by +6.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `4e088d1`	merge: `87cf12c`	diff	diff %
ydbd size	4 158 100 912 Bytes	4 158 107 136 Bytes	+6.1 KiB	+0.000%
ydbd stripped size	1 443 031 000 Bytes	1 443 032 024 Bytes	+1.0 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-06-02T18:15:04Z

⚪ 2025-06-02 18:15:03 UTC Pre-commit check linux-x86_64-relwithdebinfo for 87cf12c has started.
⚪ 2025-06-02 18:15:15 UTC Artifacts will be uploaded here
⚪ 2025-06-02 18:18:20 UTC ya make is running...
🟡 2025-06-02 19:32:07 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
21787	20396	0	5	1349	37

⚪ 2025-06-02 19:33:53 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-06-02 19:49:10 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
1130 (only retried tests)	1102	0	1	0	27

⚪ 2025-06-02 19:49:24 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-06-02 20:00:22 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
428 (only retried tests)	403	0	0	0	25

🟢 2025-06-02 20:00:30 UTC Build successful.
🟢 2025-06-02 20:00:52 UTC ydbd size 2.2 GiB changed* by +4.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `4e088d1`	merge: `87cf12c`	diff	diff %
ydbd size	2 364 212 888 Bytes	2 364 217 184 Bytes	+4.2 KiB	+0.000%
ydbd stripped size	496 309 512 Bytes	496 310 216 Bytes	+704 Bytes	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

kungasc

InitSingleKMeans: метод абсолютно confusing, не понятно ни что он делает ни как ведёт себя

проверка что все списки шардов пустые --- надо её как-то понятно назвать и вынести вверх (вызывать InitSingleKMeans только если да)
Cluster2Shards.lower_bound + InvalidShardIdx --- какая-то магия, надо сделать понятный метод возращающий список шардов по parent
AddShard зачем принимает статус, пусть проверяет внутри
то что кластер не добавляется если у него больше одного шарда никак не понять не вычитывая код пол дня

предлагаю сделать явные проверки на это в местах вызова этой функции
итого эта функция кажется совсем не нужна и переписывается в набор ифов там где мы делаем переходы

InitMultiKMeans: аналогично, назначение функции и что она делает не читается

цикл с формированием списка шардов надо вынести в понятную функцию (шарды у которых один кластер?)
внутри меняется KMeans.State, но не персистится, как будто в такой функции тоже состояние надо менять снаружи её

~~скорее всего~~ получается эта функция не нужна, как и InitSingleKMeans

vitalif · 2025-06-03T11:08:45Z

InitSingleKMeans:, InitMultiKMeans:

Упростил как обсудили вчера, также упростил Cluster2Shards :-) должно быть теперь понятно

github-actions · 2025-06-03T11:11:26Z

⚪ 2025-06-03 11:11:25 UTC Pre-commit check linux-x86_64-relwithdebinfo for bcb3381 has started.
⚪ 2025-06-03 11:11:37 UTC Artifacts will be uploaded here
⚪ 2025-06-03 11:14:44 UTC ya make is running...
⚫ 2025-06-03 11:50:26 UTC Check cancelled

github-actions · 2025-06-03T11:12:12Z

⚪ 2025-06-03 11:12:12 UTC Pre-commit check linux-x86_64-release-asan for bcb3381 has started.
⚪ 2025-06-03 11:12:22 UTC Artifacts will be uploaded here
⚪ 2025-06-03 11:15:24 UTC ya make is running...
⚫ 2025-06-03 11:50:24 UTC Check cancelled

ydb/core/tx/schemeshard/schemeshard_build_index__progress.cpp

kungasc

поправить чуть ещё

github-actions · 2025-06-03T12:03:33Z

⚪ 2025-06-03 12:03:32 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8678728 has started.
⚪ 2025-06-03 12:03:43 UTC Artifacts will be uploaded here
⚪ 2025-06-03 12:06:50 UTC ya make is running...

github-actions · 2025-06-03T12:03:34Z

⚪ 2025-06-03 12:03:33 UTC Pre-commit check linux-x86_64-release-asan for 8678728 has started.
⚪ 2025-06-03 12:03:44 UTC Artifacts will be uploaded here
⚪ 2025-06-03 12:07:00 UTC ya make is running...
⚫ 2025-06-03 13:17:43 UTC Check cancelled

…ocal}

vitalif · 2025-06-03T13:17:53Z

Тестик reboots упал, пришлось 1 строчку поправить) позеленеет и попрошу перелайкать

github-actions · 2025-06-03T13:25:36Z

⚪ 2025-06-03 16:37:45 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2136eb9 has started.
⚪ 2025-06-03 16:37:57 UTC Artifacts will be uploaded here
⚪ 2025-06-03 16:41:01 UTC ya make is running...
🟡 2025-06-03 17:48:02 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
21794	20390	0	6	1359	39

⚪ 2025-06-03 17:49:52 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-06-03 18:14:56 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
1239 (only retried tests)	1198	0	0	11	30

🟢 2025-06-03 18:15:07 UTC Build successful.
🟢 2025-06-03 18:15:23 UTC ydbd size 2.2 GiB changed* by -1.7 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `33b7f67`	merge: `2136eb9`	diff	diff %
ydbd size	2 364 337 896 Bytes	2 364 336 144 Bytes	-1.7 KiB	-0.000%
ydbd stripped size	496 324 520 Bytes	496 323 496 Bytes	-1.0 KiB	-0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-06-03T13:26:31Z

⚪ 2025-06-03 13:26:30 UTC Pre-commit check linux-x86_64-release-asan for 2136eb9 has started.
⚪ 2025-06-03 13:26:53 UTC Artifacts will be uploaded here
⚪ 2025-06-03 13:30:25 UTC ya make is running...
🟡 2025-06-03 15:16:35 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13808	13488	0	187	111	22

⚪ 2025-06-03 15:17:46 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-06-03 15:56:23 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
2136 (only retried tests)	2051	0	60	6	19

⚪ 2025-06-03 15:56:43 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-06-03 16:27:58 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
1423 (only retried tests)	1338	0	51	17	17

🟢 2025-06-03 16:28:13 UTC Build successful.
🟢 2025-06-03 16:28:48 UTC ydbd size 3.9 GiB changed* by -2.4 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `33b7f67`	merge: `2136eb9`	diff	diff %
ydbd size	4 158 103 776 Bytes	4 158 101 352 Bytes	-2.4 KiB	-0.000%
ydbd stripped size	1 442 980 216 Bytes	1 442 978 936 Bytes	-1.2 KiB	-0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

…form#19128)

vitalif requested a review from a team as a code owner June 2, 2025 08:02

github-actions bot added the not-for-changelog label Jun 2, 2025

vitalif linked an issue Jun 2, 2025 that may be closed by this pull request

Refactor FillVectorIndex "state-machine" #18579

Closed

vitalif requested a review from kungasc June 2, 2025 10:19

vitalif force-pushed the refactor-fillvectorindex branch from a418b19 to 4f0f917 Compare June 2, 2025 13:30

Refactor FillVectorIndex state machine (ydb-platform#18579)

82c36dc

vitalif force-pushed the refactor-fillvectorindex branch from 4f0f917 to 82c36dc Compare June 2, 2025 18:11

kungasc suggested changes Jun 2, 2025

View reviewed changes

vitalif requested a review from kungasc June 3, 2025 11:32

kungasc approved these changes Jun 3, 2025

View reviewed changes

kungasc suggested changes Jun 3, 2025

View reviewed changes

vitalif force-pushed the refactor-fillvectorindex branch 2 times, most recently from 0b00b2d to 6aa2600 Compare June 3, 2025 11:56

kungasc previously approved these changes Jun 3, 2025

View reviewed changes

CyberROFL previously approved these changes Jun 3, 2025

View reviewed changes

Simplify Cluster2Shards, replace Init{Single,Multi} with Add{Global,L…

665208d

…ocal}

vitalif dismissed stale reviews from CyberROFL and kungasc via 665208d June 3, 2025 13:17

vitalif force-pushed the refactor-fillvectorindex branch from 6aa2600 to 665208d Compare June 3, 2025 13:17

kungasc approved these changes Jun 3, 2025

View reviewed changes

CyberROFL approved these changes Jun 3, 2025

View reviewed changes

vitalif merged commit a836d8f into ydb-platform:main Jun 3, 2025
16 of 17 checks passed

azevaykin pushed a commit to azevaykin/ydb that referenced this pull request Jun 5, 2025

Refactor FillVectorIndex state machine (ydb-platform#18579) (ydb-plat…

e75bbe8

…form#19128)

StekPerepolnen mentioned this pull request Sep 15, 2025

audit fault stream nb 25 1 #24990

Closed

Refactor FillVectorIndex state machine (#18579) #19128

Refactor FillVectorIndex state machine (#18579) #19128

Uh oh!

Conversation

vitalif commented Jun 2, 2025

Changelog category

Uh oh!

github-actions bot commented Jun 2, 2025

Uh oh!

github-actions bot commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kungasc left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vitalif commented Jun 3, 2025

Uh oh!

github-actions bot commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kungasc left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vitalif commented Jun 3, 2025

Uh oh!

github-actions bot commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Jun 2, 2025 •

edited

Loading

github-actions bot commented Jun 2, 2025 •

edited

Loading

github-actions bot commented Jun 2, 2025 •

edited

Loading

github-actions bot commented Jun 2, 2025 •

edited

Loading

github-actions bot commented Jun 2, 2025 •

edited

Loading

github-actions bot commented Jun 2, 2025 •

edited

Loading

kungasc left a comment •

edited

Loading

github-actions bot commented Jun 3, 2025 •

edited

Loading

github-actions bot commented Jun 3, 2025 •

edited

Loading

github-actions bot commented Jun 3, 2025 •

edited

Loading

github-actions bot commented Jun 3, 2025 •

edited

Loading

github-actions bot commented Jun 3, 2025 •

edited

Loading

github-actions bot commented Jun 3, 2025 •

edited

Loading