Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](local exchange) Fix unbalance data distribution #44421

Merged
merged 2 commits into from
Nov 22, 2024

Conversation

Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Nov 21, 2024

What problem does this PR solve?

Follow-up for #44137

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.04% (9902/26032)
Line Coverage: 29.21% (82818/283514)
Region Coverage: 28.35% (42542/150082)
Branch Coverage: 24.91% (21566/86590)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1cb462f9bdda38e5f4d0203b382feb30d7eb9d49_1cb462f9bdda38e5f4d0203b382feb30d7eb9d49/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40236 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1cb462f9bdda38e5f4d0203b382feb30d7eb9d49, data reload: false

------ Round 1 ----------------------------------
q1	17589	7538	7336	7336
q2	2066	179	170	170
q3	10855	1130	1166	1130
q4	10559	765	674	674
q5	7629	2757	2743	2743
q6	248	151	151	151
q7	979	638	627	627
q8	9234	1888	1931	1888
q9	6568	6433	6436	6433
q10	7007	2310	2345	2310
q11	468	265	262	262
q12	419	229	227	227
q13	17774	3046	3061	3046
q14	241	210	218	210
q15	594	545	513	513
q16	664	592	591	591
q17	978	551	521	521
q18	7305	6707	6716	6707
q19	1336	1065	1061	1061
q20	483	186	178	178
q21	4242	3146	3485	3146
q22	374	313	312	312
Total cold run time: 107612 ms
Total hot run time: 40236 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7723	7720	7699	7699
q2	323	230	230	230
q3	2902	2965	3038	2965
q4	2127	1865	1857	1857
q5	5598	5666	5672	5666
q6	224	140	144	140
q7	2251	1829	1793	1793
q8	3380	3546	3568	3546
q9	8905	8863	8934	8863
q10	3638	3570	3557	3557
q11	605	503	509	503
q12	837	661	656	656
q13	10787	3250	3275	3250
q14	314	281	288	281
q15	595	517	513	513
q16	690	640	639	639
q17	1845	1643	1620	1620
q18	8428	7774	7568	7568
q19	1714	1599	1630	1599
q20	2122	1857	1972	1857
q21	5673	5609	5524	5524
q22	614	532	564	532
Total cold run time: 71295 ms
Total hot run time: 60858 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197842 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1cb462f9bdda38e5f4d0203b382feb30d7eb9d49, data reload: false

query1	1203	946	913	913
query2	6244	2129	2047	2047
query3	10915	4220	4153	4153
query4	67516	27931	23513	23513
query5	4908	460	462	460
query6	411	187	180	180
query7	5546	314	294	294
query8	313	240	219	219
query9	8514	2748	2755	2748
query10	458	252	259	252
query11	17305	15363	15837	15363
query12	155	112	104	104
query13	1457	423	405	405
query14	9962	7797	7575	7575
query15	216	192	189	189
query16	7173	494	530	494
query17	1072	582	577	577
query18	1875	322	318	318
query19	204	157	164	157
query20	119	115	113	113
query21	208	104	108	104
query22	4654	4630	4531	4531
query23	35004	34159	34300	34159
query24	5522	2546	2550	2546
query25	519	383	404	383
query26	681	146	149	146
query27	2099	291	295	291
query28	4802	2479	2481	2479
query29	708	431	429	429
query30	221	145	149	145
query31	995	835	863	835
query32	67	57	57	57
query33	402	295	288	288
query34	936	524	553	524
query35	840	776	772	772
query36	1063	953	980	953
query37	120	74	75	74
query38	4516	4430	4543	4430
query39	1495	1491	1467	1467
query40	202	106	101	101
query41	44	43	43	43
query42	117	103	101	101
query43	549	512	509	509
query44	1170	847	834	834
query45	188	172	169	169
query46	1179	728	725	725
query47	2025	1893	1941	1893
query48	426	325	313	313
query49	738	410	384	384
query50	837	399	407	399
query51	7470	7213	7061	7061
query52	98	89	85	85
query53	263	181	177	177
query54	507	396	397	396
query55	78	73	77	73
query56	266	247	227	227
query57	1300	1182	1152	1152
query58	214	209	213	209
query59	3301	2998	3009	2998
query60	265	251	248	248
query61	112	109	109	109
query62	788	660	689	660
query63	219	183	193	183
query64	1432	650	639	639
query65	3270	3250	3257	3250
query66	724	308	303	303
query67	15982	15661	15681	15661
query68	3858	558	557	557
query69	429	257	258	257
query70	1212	1154	1165	1154
query71	367	244	249	244
query72	6396	4066	3997	3997
query73	782	358	365	358
query74	9483	9061	9064	9061
query75	3391	2706	2662	2662
query76	1802	1052	1084	1052
query77	503	275	286	275
query78	10599	9458	9463	9458
query79	1545	589	577	577
query80	873	463	427	427
query81	503	244	226	226
query82	1262	116	118	116
query83	269	152	154	152
query84	289	71	75	71
query85	898	293	299	293
query86	392	307	303	303
query87	4727	4592	4633	4592
query88	3485	2255	2228	2228
query89	420	292	295	292
query90	2007	191	193	191
query91	135	102	101	101
query92	69	48	50	48
query93	1879	549	545	545
query94	801	293	293	293
query95	361	248	248	248
query96	619	281	271	271
query97	2917	2712	2724	2712
query98	218	193	195	193
query99	1807	1311	1299	1299
Total cold run time: 318480 ms
Total hot run time: 197842 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.12 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1cb462f9bdda38e5f4d0203b382feb30d7eb9d49, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.02
query3	0.23	0.07	0.07
query4	1.63	0.10	0.11
query5	0.43	0.41	0.42
query6	1.14	0.67	0.67
query7	0.02	0.02	0.02
query8	0.04	0.02	0.03
query9	0.59	0.52	0.51
query10	0.55	0.56	0.54
query11	0.14	0.10	0.11
query12	0.14	0.11	0.12
query13	0.61	0.61	0.59
query14	2.72	2.71	2.81
query15	0.90	0.82	0.82
query16	0.39	0.37	0.38
query17	0.98	1.09	1.05
query18	0.22	0.21	0.22
query19	1.91	1.86	1.95
query20	0.01	0.02	0.01
query21	15.37	0.57	0.59
query22	2.85	2.37	1.81
query23	17.05	0.91	0.92
query24	3.25	0.86	0.19
query25	0.17	0.18	0.10
query26	0.36	0.13	0.13
query27	0.05	0.04	0.04
query28	11.27	1.11	1.06
query29	12.53	3.33	3.34
query30	0.25	0.06	0.06
query31	2.85	0.39	0.38
query32	3.30	0.46	0.46
query33	3.00	3.05	3.07
query34	16.93	4.48	4.52
query35	4.63	4.57	4.55
query36	0.65	0.51	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 107.75 s
Total hot run time: 32.12 s

@Gabriel39
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40320 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 66903ba5560fd25ca0264811fac0cd9fd05537fa, data reload: false

------ Round 1 ----------------------------------
q1	17623	7537	7403	7403
q2	2048	184	172	172
q3	10635	1107	1186	1107
q4	10502	715	659	659
q5	7584	2815	2732	2732
q6	240	151	147	147
q7	1005	624	614	614
q8	9249	1880	1935	1880
q9	6583	6450	6458	6450
q10	7012	2295	2332	2295
q11	464	258	270	258
q12	416	223	225	223
q13	17773	3019	3051	3019
q14	237	214	212	212
q15	582	544	525	525
q16	675	592	583	583
q17	993	567	539	539
q18	7404	6808	6752	6752
q19	1332	997	1102	997
q20	469	192	186	186
q21	4299	3407	3255	3255
q22	386	324	312	312
Total cold run time: 107511 ms
Total hot run time: 40320 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7296	7281	7311	7281
q2	332	234	236	234
q3	2902	2906	3003	2906
q4	2107	1904	1828	1828
q5	5679	5710	5757	5710
q6	229	143	142	142
q7	2291	1846	1852	1846
q8	3446	3618	3585	3585
q9	8993	8967	8972	8967
q10	3651	3616	3580	3580
q11	610	511	514	511
q12	898	690	618	618
q13	10847	3255	3244	3244
q14	307	277	292	277
q15	586	529	530	529
q16	697	647	675	647
q17	1882	1631	1610	1610
q18	8455	7971	7707	7707
q19	2020	1590	1507	1507
q20	2172	1877	1881	1877
q21	5657	5611	5430	5430
q22	641	564	567	564
Total cold run time: 71698 ms
Total hot run time: 60600 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.32% (9979/26044)
Line Coverage: 29.43% (83498/283754)
Region Coverage: 28.56% (42932/150301)
Branch Coverage: 25.15% (21812/86718)
Coverage Report: http://coverage.selectdb-in.cc/coverage/66903ba5560fd25ca0264811fac0cd9fd05537fa_66903ba5560fd25ca0264811fac0cd9fd05537fa/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 198479 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 66903ba5560fd25ca0264811fac0cd9fd05537fa, data reload: false

query1	1238	960	925	925
query2	6260	2176	2169	2169
query3	10824	4104	4061	4061
query4	68379	27330	23538	23538
query5	5038	474	454	454
query6	431	184	190	184
query7	5682	310	291	291
query8	345	238	238	238
query9	9234	2680	2674	2674
query10	461	262	257	257
query11	17443	15216	15881	15216
query12	173	106	103	103
query13	1589	428	440	428
query14	11043	7585	7948	7585
query15	228	194	199	194
query16	7127	483	497	483
query17	1244	614	603	603
query18	1838	321	324	321
query19	213	172	163	163
query20	122	123	114	114
query21	216	108	108	108
query22	4760	4563	4489	4489
query23	35088	34690	34618	34618
query24	5448	2586	2558	2558
query25	494	392	401	392
query26	674	151	161	151
query27	1820	293	298	293
query28	4302	2511	2497	2497
query29	738	429	425	425
query30	214	153	153	153
query31	1019	831	856	831
query32	70	59	58	58
query33	431	298	292	292
query34	959	536	538	536
query35	880	749	765	749
query36	1075	988	1014	988
query37	123	78	80	78
query38	4568	4370	4458	4370
query39	1505	1472	1505	1472
query40	213	102	104	102
query41	45	43	66	43
query42	109	101	98	98
query43	574	522	511	511
query44	1258	874	863	863
query45	187	170	187	170
query46	1174	744	729	729
query47	2019	1933	1933	1933
query48	453	327	325	325
query49	736	404	420	404
query50	863	413	420	413
query51	7359	7363	7196	7196
query52	104	87	94	87
query53	268	182	181	181
query54	511	410	402	402
query55	78	76	78	76
query56	270	242	252	242
query57	1380	1180	1169	1169
query58	246	220	213	213
query59	3251	3106	3062	3062
query60	275	252	250	250
query61	121	106	109	106
query62	807	664	668	664
query63	228	197	198	197
query64	1388	682	661	661
query65	3338	3191	3253	3191
query66	701	304	310	304
query67	16070	15628	15647	15628
query68	3917	585	563	563
query69	429	255	253	253
query70	1126	1171	1139	1139
query71	352	256	251	251
query72	6555	4058	4101	4058
query73	790	375	368	368
query74	10278	9014	9052	9014
query75	3441	2724	2701	2701
query76	1834	1049	1130	1049
query77	512	303	300	300
query78	10527	9476	9376	9376
query79	1853	597	625	597
query80	1159	435	450	435
query81	526	243	228	228
query82	276	118	123	118
query83	207	150	156	150
query84	283	72	75	72
query85	978	307	311	307
query86	391	306	266	266
query87	4698	4539	4627	4539
query88	3638	2300	2238	2238
query89	416	298	310	298
query90	1962	196	189	189
query91	143	107	103	103
query92	63	50	51	50
query93	2563	562	555	555
query94	820	297	287	287
query95	357	249	259	249
query96	646	288	284	284
query97	2882	2744	2674	2674
query98	212	191	186	186
query99	1701	1324	1303	1303
Total cold run time: 322857 ms
Total hot run time: 198479 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.39 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 66903ba5560fd25ca0264811fac0cd9fd05537fa, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.63	0.10	0.11
query5	0.45	0.41	0.41
query6	1.17	0.66	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.51	0.50
query10	0.54	0.56	0.55
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.62	0.61	0.60
query14	2.83	2.76	2.70
query15	0.89	0.83	0.83
query16	0.38	0.38	0.39
query17	1.08	1.03	1.07
query18	0.23	0.21	0.20
query19	1.91	1.85	1.90
query20	0.02	0.01	0.01
query21	15.35	0.57	0.56
query22	2.43	2.15	1.58
query23	16.88	0.89	0.76
query24	3.44	1.28	0.95
query25	0.29	0.24	0.17
query26	0.26	0.14	0.14
query27	0.05	0.04	0.05
query28	10.06	1.11	1.08
query29	12.61	3.21	3.17
query30	0.25	0.07	0.06
query31	2.87	0.39	0.38
query32	3.27	0.46	0.46
query33	3.08	3.02	3.04
query34	16.78	4.53	4.57
query35	4.50	4.51	4.54
query36	0.68	0.49	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.18	0.13	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.27 s
Total hot run time: 32.39 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 22, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@Gabriel39 Gabriel39 merged commit 1ffecfd into apache:master Nov 22, 2024
25 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.x dev/3.0.x-conflict reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants