Added a batching for transferring from a topic to a table #15637

nshestakov · 2025-03-12T13:13:59Z

Changelog entry

...

Changelog category

Not for changelog (changelog entry is not required)

Description for reviewers

...

github-actions · 2025-03-12T13:15:10Z

🟢 2025-03-12 13:15:09 UTC The validation of the Pull Request description is successful.

github-actions · 2025-03-12T13:17:16Z

⚪ 2025-03-12 13:17:15 UTC Pre-commit check linux-x86_64-release-asan for 670495d has started.
⚪ 2025-03-12 13:17:30 UTC Artifacts will be uploaded here
⚪ 2025-03-12 13:20:25 UTC ya make is running...
🟡 2025-03-12 14:28:11 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11954	11842	0	68	6	38

⚪ 2025-03-12 14:29:37 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-03-12 14:42:38 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
160 (only retried tests)	123	0	0	1	36

🟢 2025-03-12 14:42:49 UTC Build successful.
🟢 2025-03-12 14:43:16 UTC ydbd size 3.8 GiB changed* by -362.2 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `ecf0ad1`	merge: `670495d`	diff	diff %
ydbd size	4 044 620 984 Bytes	4 044 250 048 Bytes	-362.2 KiB	-0.009%
ydbd stripped size	1 399 433 544 Bytes	1 399 449 352 Bytes	+15.4 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-03-12T13:18:17Z

⚪ 2025-03-12 13:18:16 UTC Pre-commit check linux-x86_64-relwithdebinfo for 670495d has started.
⚪ 2025-03-12 13:18:20 UTC Artifacts will be uploaded here
⚪ 2025-03-12 13:21:05 UTC ya make is running...
🟡 2025-03-12 14:17:13 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
19432	18090	0	3	1225	114

⚪ 2025-03-12 14:19:27 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-03-12 14:36:42 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
167 (only retried tests)	61	0	0	0	106

🟢 2025-03-12 14:36:52 UTC Build successful.
🟢 2025-03-12 14:37:11 UTC ydbd size 2.1 GiB changed* by -253.3 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `eda007c`	merge: `670495d`	diff	diff %
ydbd size	2 297 951 776 Bytes	2 297 692 416 Bytes	-253.3 KiB	-0.011%
ydbd stripped size	480 968 512 Bytes	480 972 032 Bytes	+3.4 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

CyberROFL

Мне кажется, правильнее будет реализовать явный коммит для TopicReader через TEvYdbProxy::TEvCommitTopicRequest/Response.

ydb/core/tx/replication/service/worker.h

ydb/core/tx/replication/ydb_proxy/ydb_proxy.cpp

Co-authored-by: Ilnaz Nizametdinov <i.nizametdinov@gmail.com>

github-actions · 2025-03-13T05:49:06Z

⚪ 2025-03-13 05:49:05 UTC Pre-commit check linux-x86_64-relwithdebinfo for deb2667 has started.
⚪ 2025-03-13 05:49:20 UTC Artifacts will be uploaded here
⚫ 2025-03-13 05:51:31 UTC Check cancelled

github-actions · 2025-03-13T05:49:32Z

⚪ 2025-03-13 05:49:31 UTC Pre-commit check linux-x86_64-release-asan for deb2667 has started.
⚪ 2025-03-13 05:49:47 UTC Artifacts will be uploaded here
⚫ 2025-03-13 05:51:31 UTC Check cancelled

github-actions · 2025-03-13T05:55:07Z

⚪ 2025-03-13 05:55:07 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5393f1b has started.
⚪ 2025-03-13 05:55:22 UTC Artifacts will be uploaded here
⚪ 2025-03-13 05:59:47 UTC ya make is running...
🟡 2025-03-13 06:48:48 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
19433	18095	0	1	1221	116

⚪ 2025-03-13 06:50:23 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-03-13 07:01:18 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
160 (only retried tests)	56	0	1	0	103

⚪ 2025-03-13 07:01:27 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-03-13 07:11:14 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
144 (only retried tests)	43	0	0	0	101

🟢 2025-03-13 07:11:21 UTC Build successful.
🟢 2025-03-13 07:11:42 UTC ydbd size 2.1 GiB changed* by -165.2 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `81bb0df`	merge: `5393f1b`	diff	diff %
ydbd size	2 298 156 104 Bytes	2 297 986 920 Bytes	-165.2 KiB	-0.007%
ydbd stripped size	481 075 040 Bytes	481 063 712 Bytes	-11.1 KiB	-0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-03-13T05:55:24Z

⚪ 2025-03-13 05:55:24 UTC Pre-commit check linux-x86_64-release-asan for 5393f1b has started.
⚪ 2025-03-13 05:55:39 UTC Artifacts will be uploaded here
⚪ 2025-03-13 06:00:05 UTC ya make is running...
🟡 2025-03-13 07:03:57 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11954	11878	0	27	14	35

⚪ 2025-03-13 07:04:57 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-03-13 07:17:31 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
116 (only retried tests)	84	0	0	1	31

🟢 2025-03-13 07:17:38 UTC Build successful.
🟢 2025-03-13 07:18:04 UTC ydbd size 3.8 GiB changed* by -143.5 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `81bb0df`	merge: `5393f1b`	diff	diff %
ydbd size	4 044 935 224 Bytes	4 044 788 256 Bytes	-143.5 KiB	-0.004%
ydbd stripped size	1 399 611 272 Bytes	1 399 655 176 Bytes	+42.9 KiB	+0.003%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

nshestakov · 2025-03-13T05:57:05Z

Мне кажется, правильнее будет реализовать явный коммит для TopicReader через TEvYdbProxy::TEvCommitTopicRequest/Response.

Тут все не так просто:

Какую задачу я решаю в этом ПР?
Уменьшение кол-ва операции записи в таблицу. что бы запись осуществлялась не на каждый батч пришедший из топика (может быть маленьким), а на батч сформированный в TransferWriter-е. Если я получил несколько сообщений до записи в таблицу, то надо коммитить только после успешной записи в таблицу всех сразу же. А значит, когда дозапрашиваю данные из топика, то коммитить мне не надо.
Возможно я дал неудачное название полю AutoCommit в TEvPoll.

Использовать же TEvCommitTopicRequest не самое удачное решение т.к. это "диструктивная" операция - после ее выполнения будет произведена перебаланчировка сессий консьюмера. Она предназначенна для точечных разовых комитов, а не для постоянных коммитов при обработке потока.

github-actions · 2025-03-13T11:49:10Z

⚪ 2025-03-13 11:49:10 UTC Pre-commit check linux-x86_64-relwithdebinfo for 9d4071d has started.
⚪ 2025-03-13 11:49:26 UTC Artifacts will be uploaded here
⚪ 2025-03-13 11:52:21 UTC ya make is running...
🟡 2025-03-13 12:44:51 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
19440	18086	0	4	1234	116

⚪ 2025-03-13 12:46:25 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-03-13 12:58:36 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
178 (only retried tests)	69	0	1	3	105

⚪ 2025-03-13 12:58:44 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-03-13 13:12:22 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
147 (only retried tests)	40	0	1	2	104

🟢 2025-03-13 13:12:32 UTC Build successful.
🟢 2025-03-13 13:12:57 UTC ydbd size 2.1 GiB changed* by +12.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `116e217`	merge: `9d4071d`	diff	diff %
ydbd size	2 297 431 040 Bytes	2 297 443 592 Bytes	+12.3 KiB	+0.001%
ydbd stripped size	480 947 392 Bytes	480 950 528 Bytes	+3.1 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-03-13T11:49:49Z

⚪ 2025-03-13 11:49:48 UTC Pre-commit check linux-x86_64-release-asan for 9d4071d has started.
⚪ 2025-03-13 11:50:04 UTC Artifacts will be uploaded here
⚪ 2025-03-13 11:52:54 UTC ya make is running...
🟡 2025-03-13 12:56:38 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11961	11875	0	34	17	35

⚪ 2025-03-13 12:58:01 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-03-13 13:11:41 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
134 (only retried tests)	88	0	0	12	34

🟢 2025-03-13 13:11:53 UTC Build successful.
🟢 2025-03-13 13:12:23 UTC ydbd size 3.8 GiB changed* by +29.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `116e217`	merge: `9d4071d`	diff	diff %
ydbd size	4 044 212 688 Bytes	4 044 242 624 Bytes	+29.2 KiB	+0.001%
ydbd stripped size	1 399 428 680 Bytes	1 399 445 992 Bytes	+16.9 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-03-13T13:49:22Z

⚪ 2025-03-13 13:49:22 UTC Pre-commit check linux-x86_64-relwithdebinfo for d8c22a9 has started.
⚪ 2025-03-13 13:49:41 UTC Artifacts will be uploaded here
⚪ 2025-03-13 13:53:00 UTC ya make is running...
🟡 2025-03-13 14:47:44 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
19440	18101	0	1	1227	111

⚪ 2025-03-13 14:49:18 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-03-13 15:01:13 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
161 (only retried tests)	55	0	1	1	104

⚪ 2025-03-13 15:01:21 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-03-13 15:13:34 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
144 (only retried tests)	38	0	0	1	105

🟢 2025-03-13 15:13:44 UTC Build successful.
🟢 2025-03-13 15:14:05 UTC ydbd size 2.1 GiB changed* by +8.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `6238290`	merge: `d8c22a9`	diff	diff %
ydbd size	2 297 484 944 Bytes	2 297 493 928 Bytes	+8.8 KiB	+0.000%
ydbd stripped size	480 962 176 Bytes	480 964 224 Bytes	+2.0 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-03-13T13:52:15Z

⚪ 2025-03-13 13:52:15 UTC Pre-commit check linux-x86_64-release-asan for d8c22a9 has started.
⚪ 2025-03-13 13:52:47 UTC Artifacts will be uploaded here
⚪ 2025-03-13 13:56:14 UTC ya make is running...
🟡 2025-03-13 15:06:49 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11960	11874	0	31	17	38

⚪ 2025-03-13 15:07:55 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-03-13 15:20:55 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
121 (only retried tests)	83	0	0	4	34

🟢 2025-03-13 15:21:14 UTC Build successful.
🟢 2025-03-13 15:21:49 UTC ydbd size 3.8 GiB changed* by +15.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `6238290`	merge: `d8c22a9`	diff	diff %
ydbd size	4 044 320 752 Bytes	4 044 336 600 Bytes	+15.5 KiB	+0.000%
ydbd stripped size	1 399 471 560 Bytes	1 399 477 864 Bytes	+6.2 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

…rm#15637) Co-authored-by: Ilnaz Nizametdinov <i.nizametdinov@gmail.com>

Co-authored-by: Ilnaz Nizametdinov <i.nizametdinov@gmail.com>

…rm#15637) Co-authored-by: Ilnaz Nizametdinov <i.nizametdinov@gmail.com>

nshestakov added 8 commits March 10, 2025 14:23

WIP

7038a01

WIP

2e4b6d4

WIP

3d6bf72

rename

261de79

Merge branch 'main' into TR-batching

e98aafb

fix

e539367

codestyle

1f48732

fix

4356f99

nshestakov requested a review from a team as a code owner March 12, 2025 13:14

nshestakov requested a review from Enjection March 12, 2025 13:14

github-actions bot added the not-for-changelog label Mar 12, 2025

nshestakov requested a review from CyberROFL March 12, 2025 17:06

CyberROFL reviewed Mar 12, 2025

View reviewed changes

nshestakov and others added 2 commits March 13, 2025 10:45

Update ydb/core/tx/replication/service/worker.h

9761e15

Co-authored-by: Ilnaz Nizametdinov <i.nizametdinov@gmail.com>

Update ydb/core/tx/replication/service/worker.h

bf9ce95

Co-authored-by: Ilnaz Nizametdinov <i.nizametdinov@gmail.com>

fixes

98c66d0

nshestakov requested a review from CyberROFL March 13, 2025 08:05

rename

defb6fb

nshestakov removed request for CyberROFL and Enjection March 13, 2025 12:45

nshestakov requested a review from CyberROFL March 13, 2025 12:45

nshestakov enabled auto-merge (squash) March 13, 2025 12:45

nshestakov added the rebase-and-check Rebase PR with the current base branch and check label Mar 13, 2025

github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label Mar 13, 2025

CyberROFL approved these changes Mar 13, 2025

View reviewed changes

nshestakov merged commit 7f029bc into ydb-platform:main Mar 13, 2025
17 of 18 checks passed

adameat pushed a commit to adameat/ydb that referenced this pull request Mar 20, 2025

Added a batching for transferring from a topic to a table (ydb-platfo…

7c1fd16

…rm#15637) Co-authored-by: Ilnaz Nizametdinov <i.nizametdinov@gmail.com>

blinkov pushed a commit that referenced this pull request Mar 21, 2025

Added a batching for transferring from a topic to a table (#15637)

37c393c

Co-authored-by: Ilnaz Nizametdinov <i.nizametdinov@gmail.com>

Emgariko pushed a commit to Emgariko/ydb that referenced this pull request Mar 24, 2025

Added a batching for transferring from a topic to a table (ydb-platfo…

881219a

…rm#15637) Co-authored-by: Ilnaz Nizametdinov <i.nizametdinov@gmail.com>

lberserq pushed a commit to lberserq/ydb that referenced this pull request Mar 28, 2025

Added a batching for transferring from a topic to a table (ydb-platfo…

1f30ec0

…rm#15637) Co-authored-by: Ilnaz Nizametdinov <i.nizametdinov@gmail.com>

nshestakov deleted the TR-batching branch April 2, 2025 09:01

nshestakov added a commit to nshestakov/ydb that referenced this pull request May 5, 2025

Added a batching for transferring from a topic to a table (ydb-platfo…

ec79acc

…rm#15637) Co-authored-by: Ilnaz Nizametdinov <i.nizametdinov@gmail.com>

StekPerepolnen mentioned this pull request Sep 15, 2025

audit fault stream nb 25 1 #24990

Closed

Added a batching for transferring from a topic to a table #15637

Added a batching for transferring from a topic to a table #15637

Uh oh!

Conversation

nshestakov commented Mar 12, 2025

Changelog entry

Changelog category

Description for reviewers

Uh oh!

github-actions bot commented Mar 12, 2025

Uh oh!

github-actions bot commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CyberROFL left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nshestakov commented Mar 13, 2025

Uh oh!

github-actions bot commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions bot commented Mar 12, 2025 •

edited

Loading

github-actions bot commented Mar 12, 2025 •

edited

Loading

github-actions bot commented Mar 13, 2025 •

edited

Loading

github-actions bot commented Mar 13, 2025 •

edited

Loading

github-actions bot commented Mar 13, 2025 •

edited

Loading

github-actions bot commented Mar 13, 2025 •

edited

Loading

github-actions bot commented Mar 13, 2025 •

edited

Loading

github-actions bot commented Mar 13, 2025 •

edited

Loading

github-actions bot commented Mar 13, 2025 •

edited

Loading

github-actions bot commented Mar 13, 2025 •

edited

Loading