Skip to content

Conversation

@Mryange
Copy link
Contributor

@Mryange Mryange commented Apr 30, 2025

What problem does this PR solve?

#50113

transmit_block should be marked as done when transmitting the last block.
Otherwise, the previous block might reach a memory limit and set done to null.

F20250430 11:49:29.413803 2176884 vdata_stream_recvr.cpp:200] Check failed: *done != nullptr 
*** Check failure stack trace: ***
    @     0x558196be1956  google::LogMessage::SendToLog()
    @     0x558196bde3a0  google::LogMessage::Flush()
    @     0x558196be2199  google::LogMessageFatal::~LogMessageFatal()
    @     0x558193b1659c  doris::vectorized::VDataStreamRecvr::SenderQueue::add_block()
    @     0x558193b1f7b5  doris::vectorized::VDataStreamRecvr::add_block()
    @     0x558193af6cf8  doris::vectorized::VDataStreamMgr::transmit_block()
    @     0x558157f7a685  doris::pipeline::DataStreamRecvrTest_transmit_block_Test::TestBody()
    @     0x5581973c8c0b  testing::internal::HandleSehExceptionsInMethodIfSupported<>()
    @     0x5581973c2a69  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @     0x55819739943a  testing::Test::Run()
    @     0x558197399e5e  testing::TestInfo::Run()
    @     0x55819739a71e  testing::TestSuite::Run()
    @     0x5581973a9dde  testing::internal::UnitTestImpl::RunAllTests()
    @     0x5581973c9a56  testing::internal::HandleSehExceptionsInMethodIfSupported<>()
    @     0x5581973c3a61  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @     0x5581973a85d3  testing::UnitTest::Run()
    @     0x5581593d8653  RUN_ALL_TESTS()

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Apr 30, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Mryange
Copy link
Contributor Author

Mryange commented Apr 30, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33877 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 70752bffbecb7fccd4408dbb9cb5feda2219b0d8, data reload: false

------ Round 1 ----------------------------------
q1	26085	5079	4993	4993
q2	2055	268	194	194
q3	10549	1274	699	699
q4	10290	1018	542	542
q5	8590	2444	2343	2343
q6	183	161	132	132
q7	930	745	604	604
q8	9313	1272	1037	1037
q9	6887	5143	5041	5041
q10	6804	2294	1875	1875
q11	467	277	283	277
q12	339	355	219	219
q13	17858	3744	3118	3118
q14	227	228	217	217
q15	540	491	481	481
q16	428	423	376	376
q17	581	863	370	370
q18	7606	7181	7150	7150
q19	1655	960	558	558
q20	327	334	220	220
q21	3904	3328	2439	2439
q22	1052	1027	992	992
Total cold run time: 116670 ms
Total hot run time: 33877 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5141	5107	5036	5036
q2	233	328	235	235
q3	2164	2646	2303	2303
q4	1329	1773	1464	1464
q5	4563	4408	4352	4352
q6	214	166	132	132
q7	1981	1896	1752	1752
q8	2584	2495	2556	2495
q9	7163	7170	7180	7170
q10	3033	3151	2751	2751
q11	563	497	493	493
q12	646	787	630	630
q13	3491	3853	3283	3283
q14	288	310	261	261
q15	523	481	485	481
q16	433	494	436	436
q17	1158	1505	1380	1380
q18	7796	7435	7376	7376
q19	790	755	881	755
q20	1982	2003	1853	1853
q21	5093	4562	4603	4562
q22	1045	1028	982	982
Total cold run time: 52213 ms
Total hot run time: 50182 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185481 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 70752bffbecb7fccd4408dbb9cb5feda2219b0d8, data reload: false

query1	1014	471	492	471
query2	6570	1851	1798	1798
query3	6744	228	220	220
query4	26667	23788	22982	22982
query5	4315	628	487	487
query6	311	220	195	195
query7	4635	488	277	277
query8	324	246	239	239
query9	8629	2598	2589	2589
query10	471	306	247	247
query11	15599	14912	14879	14879
query12	167	111	106	106
query13	1641	503	399	399
query14	9387	6092	6076	6076
query15	203	194	168	168
query16	7257	636	499	499
query17	1168	715	550	550
query18	1971	391	284	284
query19	181	179	148	148
query20	117	119	117	117
query21	207	122	103	103
query22	4033	4070	4092	4070
query23	33855	32876	33135	32876
query24	8414	2382	2389	2382
query25	549	449	400	400
query26	1237	264	168	168
query27	2750	488	328	328
query28	4382	2112	2074	2074
query29	772	544	415	415
query30	285	215	192	192
query31	928	855	749	749
query32	73	65	63	63
query33	562	358	319	319
query34	788	845	508	508
query35	773	837	728	728
query36	959	992	885	885
query37	116	111	75	75
query38	4105	4186	4087	4087
query39	1474	1409	1384	1384
query40	217	132	113	113
query41	64	58	58	58
query42	122	111	108	108
query43	503	533	474	474
query44	1276	795	795	795
query45	188	171	171	171
query46	833	1027	637	637
query47	1797	1813	1750	1750
query48	393	414	303	303
query49	827	547	451	451
query50	639	678	418	418
query51	4102	4085	4024	4024
query52	107	110	107	107
query53	221	253	183	183
query54	604	573	514	514
query55	87	84	85	84
query56	345	302	280	280
query57	1142	1116	1096	1096
query58	255	260	251	251
query59	2613	2724	2600	2600
query60	335	310	292	292
query61	128	128	130	128
query62	887	751	666	666
query63	230	180	183	180
query64	4389	1005	728	728
query65	4355	4250	4288	4250
query66	1142	407	307	307
query67	15770	15529	15306	15306
query68	8594	877	532	532
query69	458	296	265	265
query70	1175	1115	1167	1115
query71	468	315	281	281
query72	5731	4720	4740	4720
query73	711	602	348	348
query74	8910	9073	8780	8780
query75	4035	3190	2739	2739
query76	3690	1184	750	750
query77	776	382	276	276
query78	10064	10075	9292	9292
query79	2458	821	565	565
query80	713	512	467	467
query81	493	264	215	215
query82	456	127	100	100
query83	290	259	239	239
query84	295	114	94	94
query85	791	350	320	320
query86	407	299	273	273
query87	4537	4428	4371	4371
query88	3587	2213	2220	2213
query89	386	314	288	288
query90	1876	214	215	214
query91	143	149	108	108
query92	73	61	59	59
query93	1782	958	583	583
query94	668	384	314	314
query95	377	293	278	278
query96	489	584	285	285
query97	3174	3258	3104	3104
query98	241	210	202	202
query99	1429	1406	1280	1280
Total cold run time: 276481 ms
Total hot run time: 185481 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.52 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 70752bffbecb7fccd4408dbb9cb5feda2219b0d8, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.11
query3	0.25	0.20	0.19
query4	1.59	0.19	0.20
query5	0.61	0.58	0.58
query6	1.18	0.72	0.71
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.53	0.51
query10	0.57	0.57	0.56
query11	0.16	0.11	0.12
query12	0.15	0.11	0.12
query13	0.61	0.61	0.60
query14	0.78	0.82	0.81
query15	0.88	0.85	0.87
query16	0.38	0.38	0.39
query17	1.04	1.01	1.01
query18	0.21	0.20	0.20
query19	1.94	1.84	1.78
query20	0.01	0.01	0.02
query21	15.44	0.90	0.54
query22	0.74	1.24	0.62
query23	14.91	1.41	0.65
query24	7.58	1.04	1.15
query25	0.47	0.22	0.13
query26	0.65	0.15	0.14
query27	0.05	0.06	0.05
query28	9.53	0.88	0.46
query29	12.53	4.02	3.26
query30	0.25	0.11	0.07
query31	2.82	0.60	0.39
query32	3.23	0.55	0.46
query33	3.04	3.14	3.06
query34	15.89	5.17	4.51
query35	4.56	4.55	4.55
query36	0.66	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 104 s
Total hot run time: 29.52 s

@Mryange
Copy link
Contributor Author

Mryange commented Apr 30, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34029 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit db47aafde452ec97ef481cd6e940de45c51cb6c5, data reload: false

------ Round 1 ----------------------------------
q1	25955	5163	5029	5029
q2	2061	265	179	179
q3	10420	1213	698	698
q4	10227	991	558	558
q5	7543	2363	2340	2340
q6	181	163	135	135
q7	925	788	618	618
q8	9308	1302	1088	1088
q9	6845	5151	5157	5151
q10	6997	2302	1914	1914
q11	519	287	287	287
q12	353	374	222	222
q13	17771	3735	3085	3085
q14	225	226	222	222
q15	536	491	486	486
q16	420	435	379	379
q17	596	860	350	350
q18	7497	7220	7059	7059
q19	1226	943	557	557
q20	333	330	225	225
q21	3943	3353	2495	2495
q22	1044	1002	952	952
Total cold run time: 114925 ms
Total hot run time: 34029 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5150	5086	5101	5086
q2	244	325	226	226
q3	2155	2650	2259	2259
q4	1364	1771	1301	1301
q5	4485	4427	4417	4417
q6	216	171	132	132
q7	1990	1917	1775	1775
q8	2620	2572	2529	2529
q9	7178	7233	7103	7103
q10	3009	3199	2737	2737
q11	573	509	486	486
q12	670	779	627	627
q13	3570	3921	3307	3307
q14	284	314	289	289
q15	530	488	473	473
q16	453	482	453	453
q17	1188	1561	1408	1408
q18	7733	7655	7409	7409
q19	849	846	1031	846
q20	1992	2017	1823	1823
q21	5130	4806	4856	4806
q22	1116	1064	1035	1035
Total cold run time: 52499 ms
Total hot run time: 50527 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193449 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit db47aafde452ec97ef481cd6e940de45c51cb6c5, data reload: false

query1	1431	1071	1052	1052
query2	6194	1819	1806	1806
query3	11150	4593	4667	4593
query4	53698	25483	23444	23444
query5	5047	509	461	461
query6	334	199	192	192
query7	4877	519	294	294
query8	319	263	240	240
query9	5515	2572	2581	2572
query10	426	322	272	272
query11	15021	15063	14870	14870
query12	160	120	109	109
query13	1056	520	413	413
query14	10243	6424	6516	6424
query15	205	187	184	184
query16	7120	662	473	473
query17	1071	721	583	583
query18	1521	413	305	305
query19	195	191	167	167
query20	130	129	124	124
query21	198	128	112	112
query22	4427	4399	4333	4333
query23	34267	33372	33411	33372
query24	6590	2481	2420	2420
query25	469	482	401	401
query26	680	275	154	154
query27	2461	507	352	352
query28	2958	2122	2113	2113
query29	597	579	439	439
query30	273	227	192	192
query31	905	865	782	782
query32	77	67	63	63
query33	446	368	314	314
query34	787	861	531	531
query35	807	836	778	778
query36	945	1018	932	932
query37	111	101	75	75
query38	4182	4222	4186	4186
query39	1521	1424	1439	1424
query40	211	115	106	106
query41	60	52	51	51
query42	120	108	113	108
query43	516	505	492	492
query44	1326	819	816	816
query45	185	174	168	168
query46	862	1034	671	671
query47	1839	1864	1829	1829
query48	392	426	314	314
query49	684	525	425	425
query50	669	690	416	416
query51	4201	4257	4218	4218
query52	136	102	102	102
query53	227	263	187	187
query54	594	582	513	513
query55	80	86	88	86
query56	308	296	295	295
query57	1175	1208	1136	1136
query58	259	253	270	253
query59	2749	2775	2622	2622
query60	332	323	322	322
query61	135	129	126	126
query62	755	764	659	659
query63	226	200	190	190
query64	1456	1107	816	816
query65	4464	4354	4291	4291
query66	716	395	299	299
query67	15823	15603	15490	15490
query68	6175	881	523	523
query69	529	309	263	263
query70	1187	1160	1109	1109
query71	445	317	297	297
query72	5860	4852	4893	4852
query73	1335	679	353	353
query74	9014	9105	8939	8939
query75	3330	3219	2699	2699
query76	3816	1195	755	755
query77	547	372	285	285
query78	9961	10157	9281	9281
query79	2223	826	573	573
query80	600	532	441	441
query81	494	251	220	220
query82	446	136	105	105
query83	254	257	236	236
query84	291	107	94	94
query85	785	351	310	310
query86	400	307	294	294
query87	4417	4389	4353	4353
query88	3650	2263	2273	2263
query89	414	328	295	295
query90	1868	212	222	212
query91	146	140	111	111
query92	74	59	57	57
query93	1859	957	587	587
query94	710	400	295	295
query95	372	285	288	285
query96	490	561	275	275
query97	3158	3237	3085	3085
query98	249	208	199	199
query99	1677	1412	1251	1251
Total cold run time: 295690 ms
Total hot run time: 193449 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.58 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit db47aafde452ec97ef481cd6e940de45c51cb6c5, data reload: false

query1	0.04	0.04	0.03
query2	0.14	0.10	0.11
query3	0.26	0.18	0.19
query4	1.59	0.19	0.18
query5	0.61	0.60	0.59
query6	1.20	0.73	0.72
query7	0.03	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.52	0.50
query10	0.58	0.59	0.56
query11	0.15	0.11	0.11
query12	0.15	0.11	0.11
query13	0.61	0.59	0.60
query14	0.77	0.81	0.80
query15	0.89	0.85	0.85
query16	0.40	0.38	0.40
query17	1.02	1.04	1.05
query18	0.22	0.20	0.20
query19	1.91	1.79	1.84
query20	0.02	0.01	0.01
query21	15.40	0.93	0.53
query22	0.76	1.12	0.67
query23	15.01	1.39	0.66
query24	6.77	1.56	1.02
query25	0.48	0.16	0.11
query26	0.68	0.17	0.15
query27	0.05	0.05	0.05
query28	9.40	0.87	0.42
query29	12.54	3.92	3.25
query30	0.25	0.10	0.06
query31	2.81	0.62	0.39
query32	3.23	0.54	0.46
query33	2.99	3.17	3.17
query34	15.79	5.15	4.48
query35	4.53	4.56	4.54
query36	0.68	0.50	0.50
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.13	0.14
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 103.04 s
Total hot run time: 29.58 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 85.71% (12/14) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 54.83% (14784/26963)
Line Coverage 43.95% (129755/295236)
Region Coverage 42.66% (66233/155271)
Branch Coverage 37.26% (33441/89744)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (14/14) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.15% (21213/26466)
Line Coverage 74.27% (218908/294750)
Region Coverage 72.59% (131875/181672)
Branch Coverage 65.88% (67392/102302)

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 30, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@BiteTheDDDDt BiteTheDDDDt merged commit 75d68f6 into apache:master Apr 30, 2025
25 of 27 checks passed
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
…#50560)

### What problem does this PR solve?

apache#50113

transmit_block should be marked as done when transmitting the last
block.
Otherwise, the previous block might reach a memory limit and set done to
null.

```
F20250430 11:49:29.413803 2176884 vdata_stream_recvr.cpp:200] Check failed: *done != nullptr 
*** Check failure stack trace: ***
    @     0x558196be1956  google::LogMessage::SendToLog()
    @     0x558196bde3a0  google::LogMessage::Flush()
    @     0x558196be2199  google::LogMessageFatal::~LogMessageFatal()
    @     0x558193b1659c  doris::vectorized::VDataStreamRecvr::SenderQueue::add_block()
    @     0x558193b1f7b5  doris::vectorized::VDataStreamRecvr::add_block()
    @     0x558193af6cf8  doris::vectorized::VDataStreamMgr::transmit_block()
    @     0x558157f7a685  doris::pipeline::DataStreamRecvrTest_transmit_block_Test::TestBody()
    @     0x5581973c8c0b  testing::internal::HandleSehExceptionsInMethodIfSupported<>()
    @     0x5581973c2a69  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @     0x55819739943a  testing::Test::Run()
    @     0x558197399e5e  testing::TestInfo::Run()
    @     0x55819739a71e  testing::TestSuite::Run()
    @     0x5581973a9dde  testing::internal::UnitTestImpl::RunAllTests()
    @     0x5581973c9a56  testing::internal::HandleSehExceptionsInMethodIfSupported<>()
    @     0x5581973c3a61  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @     0x5581973a85d3  testing::UnitTest::Run()
    @     0x5581593d8653  RUN_ALL_TESTS()
```
### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants