Implement excess key columns drop for BlockMapJoinCore computation node #9036

igormunkin · 2024-09-10T17:46:53Z

This patchset implements drop of the specified key columns, to reduce excess data in the result, produced by the BlockMapJoinCore computation node. As a result of the patch, program builder handler for BlockMapJoinCore accepts the vector with the specified key columns, that match indices of the corresponding "left" wide flow fields.

It's worth mentioning why only left key columns are processed:

Right key columns are dropped by default, since they are the same as the left ones;
Right payload columns respect the following contract: input type provides only those fields, required by output wide type;
Left non-key columns also respect the aforementioned contract.

All in all, only the key columns used to process the join, but not required in the output type, should be processed by this computation node.

NB: MKQL version is not bumped, since there is no way to emit BlockMapJoinCore at the moment.

Changelog category

Improvement

github-actions · 2024-09-10T17:50:10Z

⚪ 2024-09-10 17:50:09 UTC Pre-commit check linux-x86_64-release-clang14 for bd224a9 has started.
⚪ 2024-09-10 17:50:25 UTC Artifacts will be uploaded here
⚪ 2024-09-10 17:53:22 UTC ya make is running...
🟢 2024-09-10 18:16:45 UTC Build successful.

github-actions · 2024-09-10T17:50:11Z

⚪ 2024-09-10 17:50:10 UTC Pre-commit check linux-x86_64-release-asan for bd224a9 has started.
⚪ 2024-09-10 17:51:02 UTC Artifacts will be uploaded here
⚪ 2024-09-10 17:54:05 UTC ya make is running...
🔴 2024-09-10 19:49:16 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
16462	16292	0	23	49	98

🟢 2024-09-10 19:50:40 UTC Build successful.
🟢 2024-09-10 19:51:10 UTC ydbd size 5.6 GiB changed* by +63.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `9e26730`	merge: `bd224a9`	diff	diff %
ydbd size	6 038 753 872 Bytes	6 038 818 824 Bytes	+63.4 KiB	+0.001%
ydbd stripped size	1 511 299 888 Bytes	1 511 316 144 Bytes	+15.9 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-10T17:50:37Z

⚪ 2024-09-10 17:50:36 UTC Pre-commit check linux-x86_64-relwithdebinfo for bd224a9 has started.
⚪ 2024-09-10 17:51:40 UTC Artifacts will be uploaded here
⚪ 2024-09-10 17:55:07 UTC ya make is running...
🟡 2024-09-10 19:20:44 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
78742	64697	0	163	13780	102

⚪ 2024-09-10 19:28:52 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-09-10 19:39:14 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
277 (only retried tests)	185	0	0	7	85

🟢 2024-09-10 19:39:22 UTC Build successful.
🟢 2024-09-10 19:40:02 UTC ydbd size 8.4 GiB changed* by +90.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `9e26730`	merge: `bd224a9`	diff	diff %
ydbd size	9 026 440 000 Bytes	9 026 532 536 Bytes	+90.4 KiB	+0.001%
ydbd stripped size	487 750 952 Bytes	487 753 896 Bytes	+2.9 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-11T23:31:58Z

⚪ 2024-09-11 23:31:57 UTC Pre-commit check linux-x86_64-release-clang14 for 2ea533e has started.
⚪ 2024-09-11 23:32:07 UTC Artifacts will be uploaded here
⚪ 2024-09-11 23:35:00 UTC ya make is running...
🟢 2024-09-11 23:55:59 UTC Build successful.

github-actions · 2024-09-11T23:32:04Z

⚪ 2024-09-11 23:32:03 UTC Pre-commit check linux-x86_64-release-asan for 2ea533e has started.
⚪ 2024-09-11 23:32:13 UTC Artifacts will be uploaded here
⚪ 2024-09-11 23:35:06 UTC ya make is running...
🔴 2024-09-12 01:26:37 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
16487	16329	0	21	39	98

🟢 2024-09-12 01:28:00 UTC Build successful.
🟢 2024-09-12 01:28:34 UTC ydbd size 5.6 GiB changed* by +69.0 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `9a9b2e6`	merge: `2ea533e`	diff	diff %
ydbd size	6 044 822 040 Bytes	6 044 892 736 Bytes	+69.0 KiB	+0.001%
ydbd stripped size	1 512 777 904 Bytes	1 512 799 152 Bytes	+20.8 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-11T23:32:05Z

⚪ 2024-09-11 23:32:04 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2ea533e has started.
⚪ 2024-09-11 23:32:14 UTC Artifacts will be uploaded here
⚪ 2024-09-11 23:35:13 UTC ya make is running...
🟡 2024-09-12 01:02:43 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
78817	64755	0	5	14025	32

⚪ 2024-09-12 01:10:07 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-09-12 01:17:30 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
272 (only retried tests)	178	0	0	7	87

🟢 2024-09-12 01:17:37 UTC Build successful.
🟢 2024-09-12 01:18:17 UTC ydbd size 8.4 GiB changed* by +96.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `9a9b2e6`	merge: `2ea533e`	diff	diff %
ydbd size	9 034 978 992 Bytes	9 035 077 920 Bytes	+96.6 KiB	+0.001%
ydbd stripped size	488 200 168 Bytes	488 202 728 Bytes	+2.5 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-12T11:26:49Z

⚪ 2024-09-12 11:26:49 UTC Pre-commit check linux-x86_64-relwithdebinfo for 844e5e4 has started.
⚪ 2024-09-12 11:26:58 UTC Artifacts will be uploaded here
⚪ 2024-09-12 11:29:57 UTC ya make is running...
🟡 2024-09-12 12:52:03 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
78816	64760	0	4	14025	27

⚪ 2024-09-12 12:59:50 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-09-12 13:07:53 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
267 (only retried tests)	175	0	0	7	85

🟢 2024-09-12 13:08:01 UTC Build successful.
🟢 2024-09-12 13:08:40 UTC ydbd size 8.4 GiB changed* by +96.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `1afa3ba`	merge: `844e5e4`	diff	diff %
ydbd size	9 035 158 744 Bytes	9 035 257 688 Bytes	+96.6 KiB	+0.001%
ydbd stripped size	488 206 664 Bytes	488 209 224 Bytes	+2.5 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-12T11:27:15Z

⚪ 2024-09-12 11:27:15 UTC Pre-commit check linux-x86_64-release-clang14 for 844e5e4 has started.
⚪ 2024-09-12 11:27:25 UTC Artifacts will be uploaded here
⚪ 2024-09-12 11:30:16 UTC ya make is running...
🟢 2024-09-12 11:49:04 UTC Build successful.

github-actions · 2024-09-12T11:27:26Z

⚪ 2024-09-12 11:27:25 UTC Pre-commit check linux-x86_64-release-asan for 844e5e4 has started.
⚪ 2024-09-12 11:27:35 UTC Artifacts will be uploaded here
⚪ 2024-09-12 11:30:37 UTC ya make is running...
🔴 2024-09-12 13:58:47 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
16534	16175	0	94	142	123

🟢 2024-09-12 14:00:13 UTC Build successful.
🟢 2024-09-12 14:00:52 UTC ydbd size 5.6 GiB changed* by +65.0 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `1afa3ba`	merge: `844e5e4`	diff	diff %
ydbd size	6 044 994 520 Bytes	6 045 061 120 Bytes	+65.0 KiB	+0.001%
ydbd stripped size	1 512 809 264 Bytes	1 512 826 416 Bytes	+16.8 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

ydb/library/yql/minikql/comp_nodes/mkql_block_map_join.cpp

vitstn

!

github-actions · 2024-09-12T14:36:55Z

⚪ 2024-09-12 14:36:54 UTC Pre-commit check linux-x86_64-release-asan for eef889f has started.
⚪ 2024-09-12 14:38:42 UTC Artifacts will be uploaded here
⚪ 2024-09-12 14:42:15 UTC ya make is running...
🔴 2024-09-12 16:36:18 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
16504	16328	0	33	44	99

🟢 2024-09-12 16:37:42 UTC Build successful.
🟢 2024-09-12 16:38:16 UTC ydbd size 5.6 GiB changed* by +94.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `70e5c67`	merge: `eef889f`	diff	diff %
ydbd size	6 050 846 736 Bytes	6 050 943 096 Bytes	+94.1 KiB	+0.002%
ydbd stripped size	1 514 122 672 Bytes	1 514 149 360 Bytes	+26.1 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-12T14:44:25Z

⚪ 2024-09-12 14:44:25 UTC Pre-commit check linux-x86_64-release-clang14 for eef889f has started.
⚪ 2024-09-12 14:44:28 UTC Artifacts will be uploaded here
⚪ 2024-09-12 14:47:29 UTC ya make is running...
🟢 2024-09-12 15:12:22 UTC Build successful.

github-actions · 2024-09-12T14:47:56Z

⚪ 2024-09-12 14:47:55 UTC Pre-commit check linux-x86_64-relwithdebinfo for eef889f has started.
⚪ 2024-09-12 14:47:59 UTC Artifacts will be uploaded here
⚪ 2024-09-12 14:50:52 UTC ya make is running...
🟡 2024-09-12 16:18:14 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
78829	64934	0	2	13797	96

⚪ 2024-09-12 16:25:44 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-09-12 16:32:57 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
101 (only retried tests)	13	0	0	0	88

🟢 2024-09-12 16:33:03 UTC Build successful.
🟡 2024-09-12 16:33:40 UTC ydbd size 8.4 GiB changed* by +117.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `70e5c67`	merge: `eef889f`	diff	diff %
ydbd size	9 040 984 808 Bytes	9 041 105 120 Bytes	+117.5 KiB	+0.001%
ydbd stripped size	488 544 488 Bytes	488 548 264 Bytes	+3.7 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

igormunkin added the area/yql YQL query language issues label Sep 10, 2024

igormunkin self-assigned this Sep 10, 2024

github-actions bot added the improvement label Sep 10, 2024

igormunkin mentioned this pull request Sep 11, 2024

Refactor tests for BlockMapJoinCore computation node (again) #9111

Merged

igormunkin force-pushed the YQL-18498-block-map-join-comp-node-drop-excess-key-columns branch from ecdd0f2 to 8970a44 Compare September 11, 2024 23:28

igormunkin changed the title ~~Implement excess key columns drop for BlockMapJoin computation node~~ Implement excess key columns drop for BlockMapJoinCore computation node Sep 12, 2024

github-actions bot added improvement and removed improvement labels Sep 12, 2024

igormunkin force-pushed the YQL-18498-block-map-join-comp-node-drop-excess-key-columns branch from 8970a44 to 102ed94 Compare September 12, 2024 11:23

igormunkin marked this pull request as ready for review September 12, 2024 11:25

igormunkin requested a review from a team as a code owner September 12, 2024 11:25

igormunkin requested a review from vitstn September 12, 2024 11:25

github-actions bot added improvement and removed improvement labels Sep 12, 2024

github-actions bot removed the improvement label Sep 12, 2024

github-actions bot added the improvement label Sep 12, 2024

vitstn reviewed Sep 12, 2024

View reviewed changes

ydb/library/yql/minikql/comp_nodes/mkql_block_map_join.cpp Outdated Show resolved Hide resolved

vitstn reviewed Sep 12, 2024

View reviewed changes

ydb/library/yql/minikql/comp_nodes/mkql_block_map_join.cpp Outdated Show resolved Hide resolved

vitstn requested changes Sep 12, 2024

View reviewed changes

igormunkin added 6 commits September 12, 2024 13:35

Adjust BlockMapJoinCore program builder with key column drops

8233277

Implement BlockMapJoinCore with key column drops

675f249

Adjust test infrastructure for key drops

508e0da

Add tests

b04f134

s/TSet/THashSet/gc

36ab859

Use I/O mapping to produce tight loop

0cb5832

igormunkin force-pushed the YQL-18498-block-map-join-comp-node-drop-excess-key-columns branch from 102ed94 to 0cb5832 Compare September 12, 2024 14:08

igormunkin requested a review from vitstn September 12, 2024 14:10

vitstn approved these changes Sep 12, 2024

View reviewed changes

igormunkin merged commit 4f855a7 into ydb-platform:main Sep 12, 2024
10 of 12 checks passed

igormunkin deleted the YQL-18498-block-map-join-comp-node-drop-excess-key-columns branch September 12, 2024 19:46

This was referenced Sep 13, 2024

YQ CTAS fix for script executions #9228

Merged

YQ fix cmake library name clash #9235

Merged

StekPerepolnen mentioned this pull request Sep 17, 2024

doc hc api images source #9376

Merged

This was referenced Sep 18, 2024

YQ kqprun added recipe #9453

Merged

YQ-3684 fix screts not found error handling #9501

Merged

shnikd mentioned this pull request Sep 19, 2024

Fix pgwire auth #9531

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement excess key columns drop for BlockMapJoinCore computation node #9036

Implement excess key columns drop for BlockMapJoinCore computation node #9036

igormunkin commented Sep 10, 2024 •

edited

Loading

github-actions bot commented Sep 10, 2024 •

edited

Loading

github-actions bot commented Sep 10, 2024 •

edited

Loading

github-actions bot commented Sep 10, 2024 •

edited

Loading

github-actions bot commented Sep 11, 2024 •

edited

Loading

github-actions bot commented Sep 11, 2024 •

edited

Loading

github-actions bot commented Sep 11, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading

vitstn left a comment

github-actions bot commented Sep 12, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading

Implement excess key columns drop for BlockMapJoinCore computation node #9036

Implement excess key columns drop for BlockMapJoinCore computation node #9036

Conversation

igormunkin commented Sep 10, 2024 • edited Loading

Changelog category

github-actions bot commented Sep 10, 2024 • edited Loading

github-actions bot commented Sep 10, 2024 • edited Loading

github-actions bot commented Sep 10, 2024 • edited Loading

github-actions bot commented Sep 11, 2024 • edited Loading

github-actions bot commented Sep 11, 2024 • edited Loading

github-actions bot commented Sep 11, 2024 • edited Loading

github-actions bot commented Sep 12, 2024 • edited Loading

github-actions bot commented Sep 12, 2024 • edited Loading

github-actions bot commented Sep 12, 2024 • edited Loading

vitstn left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 12, 2024 • edited Loading

github-actions bot commented Sep 12, 2024 • edited Loading

github-actions bot commented Sep 12, 2024 • edited Loading

igormunkin commented Sep 10, 2024 •

edited

Loading

github-actions bot commented Sep 10, 2024 •

edited

Loading

github-actions bot commented Sep 10, 2024 •

edited

Loading

github-actions bot commented Sep 10, 2024 •

edited

Loading

github-actions bot commented Sep 11, 2024 •

edited

Loading

github-actions bot commented Sep 11, 2024 •

edited

Loading

github-actions bot commented Sep 11, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024 •

edited

Loading