Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bugfix](core) runtime state is destroyed when spill task is run #39566

Merged
merged 1 commit into from
Aug 20, 2024

Conversation

yiguolei
Copy link
Contributor

==17355==ERROR: AddressSanitizer: heap-use-after-free on address 0x61c00388c269 at pc 0x55b47f79e5c4 bp 0x7fde7dd98e30 sp 0x7fde7dd98e28
READ of size 1 at 0x61c00388c269 thread T604 (SpillIOThreadPo)
#0 0x55b47f79e5c3 in doris::QueryContext::is_nereids() const /root/doris/be/src/runtime/query_context.h:229:38
#1 0x55b47f79e5c3 in doris::RuntimeState::is_nereids() const /root/doris/be/src/runtime/runtime_state.cpp:542:24
#2 0x55b47f89afed in doris::AttachTask::AttachTask(doris::RuntimeState*) /root/doris/be/src/runtime/thread_context.cpp:48:50
#3 0x55b4b039e7e5 in doris::pipeline::SpillRunnable::run() /root/doris/be/src/pipeline/exec/spill_utils.h:43:9
#4 0x55b47fdf739d in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24
#5 0x55b47fdd05de in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
#6 0x55b47fdd05de in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
#7 0x7fe1bcb5e608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
#8 0x7fe1bce0b132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@yiguolei
Copy link
Contributor Author

run buildall

Copy link
Member

@mrhhsg mrhhsg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 19, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@yiguolei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38170 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8c39b4bf3db45272ceed5ce01755c6342f4c70f9, data reload: false

------ Round 1 ----------------------------------
q1	17929	4422	4267	4267
q2	2056	222	210	210
q3	10399	1147	1085	1085
q4	10155	801	709	709
q5	7799	2826	2810	2810
q6	265	158	160	158
q7	1023	662	648	648
q8	9403	2076	2084	2076
q9	7079	6533	6531	6531
q10	7087	2178	2250	2178
q11	490	274	279	274
q12	431	262	261	261
q13	17814	3028	2992	2992
q14	315	265	251	251
q15	557	529	554	529
q16	535	402	405	402
q17	968	710	697	697
q18	7594	6862	6853	6853
q19	6669	1095	974	974
q20	697	340	352	340
q21	4004	3010	2893	2893
q22	1096	1032	1032	1032
Total cold run time: 114365 ms
Total hot run time: 38170 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4424	4311	4297	4297
q2	396	299	307	299
q3	2816	2627	2600	2600
q4	1879	1706	1698	1698
q5	5619	5483	5389	5389
q6	233	144	147	144
q7	2109	1734	1735	1734
q8	3184	3328	3340	3328
q9	8428	8420	8401	8401
q10	3380	3180	3142	3142
q11	624	519	517	517
q12	803	626	632	626
q13	17114	3026	3014	3014
q14	320	293	296	293
q15	555	506	512	506
q16	488	440	437	437
q17	1801	1503	1465	1465
q18	7565	7618	7427	7427
q19	1691	1636	1554	1554
q20	2021	1828	1792	1792
q21	5311	5088	5012	5012
q22	1143	1041	1022	1022
Total cold run time: 71904 ms
Total hot run time: 54697 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189729 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8c39b4bf3db45272ceed5ce01755c6342f4c70f9, data reload: false

query1	961	395	382	382
query2	6718	1927	1919	1919
query3	6666	224	246	224
query4	34313	23387	23196	23196
query5	4393	704	705	704
query6	303	212	228	212
query7	4611	329	326	326
query8	475	420	429	420
query9	8773	2523	2503	2503
query10	517	346	337	337
query11	17755	15011	14982	14982
query12	188	138	137	137
query13	1710	446	426	426
query14	9999	6634	7155	6634
query15	267	190	193	190
query16	7893	482	512	482
query17	1609	600	578	578
query18	2129	337	336	336
query19	229	177	166	166
query20	145	136	137	136
query21	245	165	141	141
query22	4360	3920	3919	3919
query23	33941	33284	33280	33280
query24	11151	2866	2926	2866
query25	629	412	417	412
query26	730	175	178	175
query27	2105	301	296	296
query28	6134	2140	2118	2118
query29	761	459	443	443
query30	338	185	184	184
query31	1076	878	837	837
query32	122	81	82	81
query33	816	340	334	334
query34	910	505	485	485
query35	895	784	787	784
query36	1089	937	976	937
query37	155	101	101	101
query38	4046	3857	3750	3750
query39	1534	1458	1478	1458
query40	245	156	156	156
query41	144	176	178	176
query42	146	128	128	128
query43	554	527	501	501
query44	1282	797	818	797
query45	237	203	239	203
query46	1135	773	740	740
query47	1937	1897	1849	1849
query48	405	328	344	328
query49	1204	576	577	576
query50	854	460	469	460
query51	6864	6795	6720	6720
query52	124	113	108	108
query53	306	226	226	226
query54	971	503	502	502
query55	94	87	90	87
query56	323	309	307	307
query57	1211	1109	1113	1109
query58	307	301	326	301
query59	3020	2775	2754	2754
query60	380	331	333	331
query61	154	147	145	145
query62	899	687	722	687
query63	261	228	222	222
query64	5053	2338	1820	1820
query65	3210	3294	3158	3158
query66	1128	671	674	671
query67	15427	15030	14956	14956
query68	6198	591	586	586
query69	778	426	345	345
query70	1199	1245	1177	1177
query71	559	312	317	312
query72	7610	2276	2080	2080
query73	839	354	354	354
query74	9200	8827	8872	8827
query75	4898	2750	2750	2750
query76	4453	1034	972	972
query77	902	454	441	441
query78	10465	9362	9027	9027
query79	8449	556	559	556
query80	1172	612	650	612
query81	622	261	263	261
query82	1326	160	159	159
query83	364	213	214	213
query84	295	97	97	97
query85	1360	354	350	350
query86	468	312	327	312
query87	4366	4244	4158	4158
query88	5143	2480	2468	2468
query89	552	329	319	319
query90	1966	230	227	227
query91	154	131	126	126
query92	86	73	76	73
query93	6330	555	542	542
query94	841	326	330	326
query95	381	288	283	283
query96	617	289	279	279
query97	3203	3066	3134	3066
query98	262	268	225	225
query99	1608	1306	1297	1297
Total cold run time: 315232 ms
Total hot run time: 189729 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.96 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8c39b4bf3db45272ceed5ce01755c6342f4c70f9, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.07	0.06
query4	1.65	0.09	0.10
query5	0.52	0.49	0.48
query6	1.14	0.72	0.72
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.56	0.49	0.50
query10	0.56	0.52	0.54
query11	0.16	0.13	0.12
query12	0.16	0.12	0.12
query13	0.61	0.60	0.59
query14	0.76	0.78	0.78
query15	0.86	0.82	0.83
query16	0.38	0.37	0.38
query17	0.95	0.98	1.00
query18	0.22	0.20	0.20
query19	1.86	1.85	1.77
query20	0.01	0.01	0.01
query21	15.43	0.67	0.67
query22	3.88	7.05	1.99
query23	18.28	1.37	1.21
query24	2.14	0.22	0.22
query25	0.15	0.10	0.08
query26	0.27	0.18	0.18
query27	0.09	0.07	0.08
query28	13.26	1.02	1.00
query29	12.63	3.37	3.26
query30	0.43	0.24	0.24
query31	2.80	0.40	0.39
query32	3.25	0.48	0.49
query33	2.93	2.98	2.93
query34	17.02	4.37	4.35
query35	4.46	4.41	4.41
query36	0.66	0.48	0.50
query37	0.21	0.17	0.18
query38	0.17	0.17	0.16
query39	0.07	0.06	0.06
query40	0.18	0.15	0.15
query41	0.11	0.07	0.07
query42	0.08	0.06	0.06
query43	0.07	0.07	0.06
Total cold run time: 109.41 s
Total hot run time: 30.96 s

@yiguolei yiguolei merged commit 37dd56c into apache:master Aug 20, 2024
25 of 28 checks passed
dataroaring pushed a commit that referenced this pull request Aug 26, 2024
)

==17355==ERROR: AddressSanitizer: heap-use-after-free on address
0x61c00388c269 at pc 0x55b47f79e5c4 bp 0x7fde7dd98e30 sp 0x7fde7dd98e28
READ of size 1 at 0x61c00388c269 thread T604 (SpillIOThreadPo)
#0 0x55b47f79e5c3 in doris::QueryContext::is_nereids() const
/root/doris/be/src/runtime/query_context.h:229:38
#1 0x55b47f79e5c3 in doris::RuntimeState::is_nereids() const
/root/doris/be/src/runtime/runtime_state.cpp:542:24
#2 0x55b47f89afed in doris::AttachTask::AttachTask(doris::RuntimeState*)
/root/doris/be/src/runtime/thread_context.cpp:48:50
#3 0x55b4b039e7e5 in doris::pipeline::SpillRunnable::run()
/root/doris/be/src/pipeline/exec/spill_utils.h:43:9
#4 0x55b47fdf739d in doris::ThreadPool::dispatch_thread()
/root/doris/be/src/util/threadpool.cpp:543:24
#5 0x55b47fdd05de in std::function<void ()>::operator()() const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
#6 0x55b47fdd05de in doris::Thread::supervise_thread(void*)
/root/doris/be/src/util/thread.cpp:498:5
#7 0x7fe1bcb5e608 in start_thread
/build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
#8 0x7fe1bce0b132 in __clone
/build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Co-authored-by: yiguolei <yiguolei@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.x dev/3.0.2-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants