Limit SelfHeal reassign requests in-flight #17618

serbel324 · 2025-04-23T13:28:20Z

Changelog entry

Don't allow SelfHeal actor to create more than 1 active ReassignerActor. This change will prevent SelfHeal from overloading BSC with ReassignItem requests thus causing DoS.

Changelog category

Improvement

github-actions · 2025-04-23T13:31:11Z

⚪ 2025-04-23 13:31:10 UTC Pre-commit check linux-x86_64-release-asan for 6c46a83 has started.
⚪ 2025-04-23 13:31:36 UTC Artifacts will be uploaded here
⚪ 2025-04-23 13:35:08 UTC ya make is running...
⚫ 2025-04-23 14:06:50 UTC Check cancelled

github-actions · 2025-04-23T13:35:55Z

🟢 2025-04-23 13:41:53 UTC The validation of the Pull Request description is successful.

github-actions · 2025-04-23T13:39:27Z

⚪ 2025-04-23 13:39:27 UTC Pre-commit check linux-x86_64-relwithdebinfo for 6c46a83 has started.
⚪ 2025-04-23 13:39:30 UTC Artifacts will be uploaded here
⚪ 2025-04-23 13:42:18 UTC ya make is running...
⚫ 2025-04-23 14:06:49 UTC Check cancelled

github-actions · 2025-04-23T14:08:45Z

⚪ 2025-04-23 14:08:44 UTC Pre-commit check linux-x86_64-relwithdebinfo for 13cbede has started.
⚪ 2025-04-23 14:08:57 UTC Artifacts will be uploaded here
⚪ 2025-04-23 14:11:39 UTC ya make is running...
⚫ 2025-04-23 14:39:29 UTC Check cancelled

github-actions · 2025-04-23T14:10:46Z

⚪ 2025-04-23 14:10:45 UTC Pre-commit check linux-x86_64-release-asan for 13cbede has started.
⚪ 2025-04-23 14:11:01 UTC Artifacts will be uploaded here
⚪ 2025-04-23 14:13:57 UTC ya make is running...
⚫ 2025-04-23 14:39:30 UTC Check cancelled

…s active already

github-actions · 2025-04-23T14:40:09Z

⚪ 2025-04-23 14:40:09 UTC Pre-commit check linux-x86_64-relwithdebinfo for 3e35e42 has started.
⚫ 2025-04-23 14:40:31 UTC Check cancelled

github-actions · 2025-04-23T14:42:22Z

⚪ 2025-04-23 14:42:21 UTC Pre-commit check linux-x86_64-release-asan for 78fce3c has started.
⚪ 2025-04-23 14:42:37 UTC Artifacts will be uploaded here
⚪ 2025-04-23 14:45:30 UTC ya make is running...
🟡 2025-04-23 16:11:19 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
12659	12507	0	80	49	23

⚪ 2025-04-23 16:12:29 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-04-23 16:26:12 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
195 (only retried tests)	113	0	42	18	22

⚪ 2025-04-23 16:26:21 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-04-23 16:39:07 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
109 (only retried tests)	44	0	38	4	23

🟢 2025-04-23 16:39:14 UTC Build successful.
🟢 2025-04-23 16:39:49 UTC ydbd size 3.8 GiB changed* by +65.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `f1d45a2`	merge: `78fce3c`	diff	diff %
ydbd size	4 133 776 496 Bytes	4 133 843 592 Bytes	+65.5 KiB	+0.002%
ydbd stripped size	1 431 224 632 Bytes	1 431 242 296 Bytes	+17.2 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-04-23T14:44:26Z

⚪ 2025-04-23 14:44:25 UTC Pre-commit check linux-x86_64-relwithdebinfo for 78fce3c has started.
⚪ 2025-04-23 14:44:34 UTC Artifacts will be uploaded here
⚪ 2025-04-23 14:47:22 UTC ya make is running...
🟡 2025-04-23 16:02:04 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
20405	19001	0	14	1284	106

⚪ 2025-04-23 16:03:49 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-04-23 16:17:24 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
194 (only retried tests)	94	0	8	0	92

⚪ 2025-04-23 16:17:33 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-04-23 16:28:16 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
136 (only retried tests)	37	0	8	0	91

🟢 2025-04-23 16:28:26 UTC Build successful.
🟢 2025-04-23 16:28:49 UTC ydbd size 2.2 GiB changed* by +38.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `f1d45a2`	merge: `78fce3c`	diff	diff %
ydbd size	2 353 148 824 Bytes	2 353 188 176 Bytes	+38.4 KiB	+0.002%
ydbd stripped size	493 969 824 Bytes	493 979 488 Bytes	+9.4 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-04-24T09:50:35Z

⚪ 2025-04-24 09:50:35 UTC Pre-commit check linux-x86_64-release-asan for a95a318 has started.
⚪ 2025-04-24 09:50:50 UTC Artifacts will be uploaded here
⚪ 2025-04-24 09:53:35 UTC ya make is running...
🟡 2025-04-24 11:09:17 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
12670	12525	0	74	48	23

⚪ 2025-04-24 11:10:22 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-04-24 11:23:53 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
183 (only retried tests)	123	0	34	4	22

⚪ 2025-04-24 11:24:01 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-04-24 11:36:02 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
82 (only retried tests)	25	0	35	1	21

🟢 2025-04-24 11:36:08 UTC Build successful.
🟡 2025-04-24 11:36:41 UTC ydbd size 3.9 GiB changed* by +285.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `0ddb5fd`	merge: `a95a318`	diff	diff %
ydbd size	4 135 474 216 Bytes	4 135 766 400 Bytes	+285.3 KiB	+0.007%
ydbd stripped size	1 431 777 912 Bytes	1 431 880 632 Bytes	+100.3 KiB	+0.007%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-04-24T11:16:01Z

⚪ 2025-04-24 11:16:00 UTC Pre-commit check linux-x86_64-relwithdebinfo for a95a318 has started.
⚪ 2025-04-24 11:16:50 UTC Artifacts will be uploaded here
⚪ 2025-04-24 11:20:04 UTC ya make is running...
🟡 2025-04-24 12:35:28 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
20419	19027	0	2	1286	104

⚪ 2025-04-24 12:37:09 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-04-24 12:49:34 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
171 (only retried tests)	76	0	0	0	95

🟢 2025-04-24 12:49:43 UTC Build successful.
🟡 2025-04-24 12:50:03 UTC ydbd size 2.2 GiB changed* by +204.1 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `305b5cd`	merge: `a95a318`	diff	diff %
ydbd size	2 354 063 736 Bytes	2 354 272 720 Bytes	+204.1 KiB	+0.009%
ydbd stripped size	494 146 784 Bytes	494 181 824 Bytes	+34.2 KiB	+0.007%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

alexvru · 2025-04-25T06:40:11Z

ydb/core/mind/bscontroller/self_heal.cpp


        void Handle(TEvReassignerDone::TPtr& ev) {
-            if (const auto it = Groups.find(ev->Get()->GroupId); it != Groups.end() && it->second.ReassignerActorId == ev->Sender) {
+            ActiveReassignerActorId = std::nullopt;


I'd check that ev->Sender is the same as this.

ydb/core/mind/bscontroller/self_heal.cpp

github-actions · 2025-04-25T08:53:16Z

⚪ 2025-04-25 08:53:16 UTC Pre-commit check linux-x86_64-relwithdebinfo for 167f173 has started.
⚪ 2025-04-25 08:53:31 UTC Artifacts will be uploaded here
⚪ 2025-04-25 08:56:27 UTC ya make is running...
🟡 2025-04-25 10:00:25 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
20421	19017	0	5	1354	45

⚪ 2025-04-25 10:02:06 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-04-25 10:03:31 UTC ydbd size 2.2 GiB changed* by +16.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `d5ac20e`	merge: `167f173`	diff	diff %
ydbd size	2 354 788 952 Bytes	2 354 806 088 Bytes	+16.7 KiB	+0.001%
ydbd stripped size	494 254 080 Bytes	494 257 792 Bytes	+3.6 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}
⚫ 2025-04-25 10:03:33 UTC Check cancelled

github-actions · 2025-04-25T08:53:18Z

⚪ 2025-04-25 08:53:17 UTC Pre-commit check linux-x86_64-release-asan for 167f173 has started.
⚪ 2025-04-25 08:53:34 UTC Artifacts will be uploaded here
⚪ 2025-04-25 08:56:30 UTC ya make is running...
⚫ 2025-04-25 10:03:15 UTC Check cancelled

github-actions · 2025-04-25T10:11:31Z

⚪ 2025-04-25 10:11:31 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5b6208e has started.
⚪ 2025-04-25 10:11:45 UTC Artifacts will be uploaded here
⚪ 2025-04-25 10:14:33 UTC ya make is running...
⚫ 2025-04-25 11:05:02 UTC Check cancelled

github-actions · 2025-04-25T10:20:41Z

⚪ 2025-04-25 10:20:41 UTC Pre-commit check linux-x86_64-release-asan for 5b6208e has started.
⚪ 2025-04-25 10:20:56 UTC Artifacts will be uploaded here
⚪ 2025-04-25 10:23:49 UTC ya make is running...
⚫ 2025-04-25 11:05:03 UTC Check cancelled

github-actions · 2025-04-25T11:06:26Z

⚪ 2025-04-25 11:06:25 UTC Pre-commit check linux-x86_64-relwithdebinfo for d28c43b has started.
⚪ 2025-04-25 11:06:41 UTC Artifacts will be uploaded here
⚪ 2025-04-25 11:09:29 UTC ya make is running...
🟡 2025-04-25 12:56:23 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
20439	19054	0	3	1342	40

⚪ 2025-04-25 12:58:04 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-04-25 13:11:32 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
95 (only retried tests)	64	0	0	0	31

🟢 2025-04-25 13:11:42 UTC Build successful.
🟢 2025-04-25 13:12:02 UTC ydbd size 2.2 GiB changed* by -12.1 MiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `db1cfac`	merge: `d28c43b`	diff	diff %
ydbd size	2 354 820 496 Bytes	2 342 117 016 Bytes	-12.1 MiB	-0.539%
ydbd stripped size	494 258 752 Bytes	492 576 448 Bytes	-1.6 MiB	-0.340%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-04-25T11:11:20Z

⚪ 2025-04-25 11:11:19 UTC Pre-commit check linux-x86_64-release-asan for d28c43b has started.
⚪ 2025-04-25 11:11:34 UTC Artifacts will be uploaded here
⚪ 2025-04-25 11:14:17 UTC ya make is running...
🟡 2025-04-25 13:13:19 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
12685	12474	0	137	49	25

⚪ 2025-04-25 13:14:23 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-04-25 13:28:09 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
253 (only retried tests)	171	0	42	19	21

⚪ 2025-04-25 13:28:21 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-04-25 13:42:12 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
104 (only retried tests)	44	0	36	3	21

🟢 2025-04-25 13:42:20 UTC Build successful.
🟢 2025-04-25 13:43:00 UTC ydbd size 3.8 GiB changed* by -15.8 MiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `d7c0986`	merge: `d28c43b`	diff	diff %
ydbd size	4 136 732 624 Bytes	4 120 166 736 Bytes	-15.8 MiB	-0.400%
ydbd stripped size	1 432 102 872 Bytes	1 428 488 504 Bytes	-3.4 MiB	-0.252%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

antonkovalenko · 2025-07-01T13:49:25Z

KIKIMR-23558

Limit in flight self-heal reassign requests by 1

661aa7b

github-actions bot added the new-feature label Apr 23, 2025

github-actions bot added improvement and removed new-feature labels Apr 23, 2025

serbel324 force-pushed the bsc/self-heal/throttle-requests branch from 403104d to f7dba39 Compare April 23, 2025 14:39

Don't allow GroupLayoutSanitizer to create ReassignerActor if there i…

a0981bd

…s active already

serbel324 force-pushed the bsc/self-heal/throttle-requests branch from f7dba39 to a0981bd Compare April 23, 2025 14:40

Fix stuck reassigns

2b62f21

This was linked to issues Apr 24, 2025

Throttle SanitizeGroup requests sent by GroupLayoutSanitizer #16726

Closed

Throttle ReassignDisk requests sent by SelfHeal background process #16727

Closed

alexvru reviewed Apr 25, 2025

View reviewed changes

alexvru requested changes Apr 25, 2025

View reviewed changes

ydb/core/mind/bscontroller/self_heal.cpp Show resolved Hide resolved

Check Reassigner ActorId

689ccb6

Remove copypaste

4709073

Don't enqueue unreassignable groups for reassign

ce0c35f

alexvru approved these changes Apr 25, 2025

View reviewed changes

serbel324 merged commit 0826f08 into ydb-platform:main Apr 25, 2025
14 checks passed

serbel324 added a commit to serbel324/ydb that referenced this pull request May 14, 2025

Limit SelfHeal reassign requests in-flight (ydb-platform#17618)

bb7c9d4

serbel324 mentioned this pull request May 15, 2025

Limit SelfHeal reassign requests in-flight (#17618) #18298

Merged

serbel324 added a commit that referenced this pull request May 15, 2025

Limit SelfHeal reassign requests in-flight (#17618) (#18298)

a05a0b9

StekPerepolnen mentioned this pull request Sep 15, 2025

audit fault stream nb 25 1 #24990

Closed

liruoko added the changelog/f25-3 PR участвует в списке изменений label Nov 19, 2025

Limit SelfHeal reassign requests in-flight #17618

Limit SelfHeal reassign requests in-flight #17618

Uh oh!

Conversation

serbel324 commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog entry

Changelog category

Uh oh!

github-actions bot commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexvru Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

antonkovalenko commented Jul 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

serbel324 commented Apr 23, 2025 •

edited

Loading

github-actions bot commented Apr 23, 2025 •

edited

Loading

github-actions bot commented Apr 23, 2025 •

edited

Loading

github-actions bot commented Apr 23, 2025 •

edited

Loading

github-actions bot commented Apr 23, 2025 •

edited

Loading

github-actions bot commented Apr 23, 2025 •

edited

Loading

github-actions bot commented Apr 23, 2025 •

edited

Loading

github-actions bot commented Apr 23, 2025 •

edited

Loading

github-actions bot commented Apr 23, 2025 •

edited

Loading

github-actions bot commented Apr 24, 2025 •

edited

Loading

github-actions bot commented Apr 24, 2025 •

edited

Loading

github-actions bot commented Apr 25, 2025 •

edited

Loading

github-actions bot commented Apr 25, 2025 •

edited

Loading

github-actions bot commented Apr 25, 2025 •

edited

Loading

github-actions bot commented Apr 25, 2025 •

edited

Loading

github-actions bot commented Apr 25, 2025 •

edited

Loading

github-actions bot commented Apr 25, 2025 •

edited

Loading