Skip to content

branch-3.0: [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. #49966 #50007

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Apr 22, 2025
Merged

branch-3.0: [fix](parquet)fix when hive_parquet_use_column_names=false && read partition tb cause be core. #49966 #50007

morningman merged 1 commit into from
Apr 22, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Apr 14, 2025

Cherry-picked from #49966

@hubgeter
[fix](parquet)fix when hive_parquet_use_column_names=false && read pa…
336c71a 
…rtition tb cause be core. (#49966)

### What problem does this PR solve?
related pr : #38432

Problem Summary:
when you query hive parquet format partition table, and `set
hive_parquet_use_column_names = false`, maybe you will get :
```
*** SIGABRT unknown detail explain (@0x2f59de) received by PID 3103198 (TID 3110278 OR 0x7f51c8e63640) from PID 3103198; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421
 1# 0x00007F55DFB45520 in /lib/x86_64-linux-gnu/libc.so.6
 2# pthread_kill at ./nptl/pthread_kill.c:89
 3# raise at ../sysdeps/posix/raise.c:27
 4# abort at ./stdlib/abort.c:81
 5# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75
 6# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
 7# 0x000055C8BD4E2041 in /mnt/disk1/doris-clusters/doris-master/output/be/lib/doris_be
 8# 0x000055C8BD4E2194 in /mnt/disk1/doris-clusters/doris-master/output/be/lib/doris_be
 9# 0x000055C8BD4E2586 in /mnt/disk1/doris-clusters/doris-master/output/be/lib/doris_be
10# std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:265
11# doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) at /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:586
````
The reason is that when `get_next_block` replaces the column name, data
out of bounds occurs.
@github-actions github-actions bot requested a review from dataroaring as a code owner April 14, 2025 01:23
@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 14, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Apr 14, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 14, 2025

TPC-H: Total hot run time: 39911 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 336c71a9988f85050563383abcf7d3120b9f241e, data reload: false

------ Round 1 ----------------------------------
q1	17565	6698	6587	6587
q2	2046	169	189	169
q3	10923	1081	1178	1081
q4	10511	723	750	723
q5	7767	2838	2867	2838
q6	228	133	133	133
q7	973	618	607	607
q8	9351	1918	1987	1918
q9	6587	6384	6507	6384
q10	6995	2260	2324	2260
q11	468	265	266	265
q12	393	206	211	206
q13	17787	2969	2957	2957
q14	246	208	210	208
q15	512	462	462	462
q16	683	594	592	592
q17	965	540	599	540
q18	7192	6619	6709	6619
q19	1398	1107	1012	1012
q20	496	206	200	200
q21	4037	3177	3155	3155
q22	1120	995	1001	995
Total cold run time: 108243 ms
Total hot run time: 39911 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6625	6577	6578	6577
q2	336	242	231	231
q3	2859	2751	2899	2751
q4	2005	1766	1825	1766
q5	5808	5759	5756	5756
q6	212	122	127	122
q7	2212	1860	1854	1854
q8	3350	3580	3500	3500
q9	8868	8825	8884	8825
q10	3535	3498	3503	3498
q11	598	493	480	480
q12	806	626	601	601
q13	9643	3204	3212	3204
q14	307	263	268	263
q15	511	461	468	461
q16	692	659	666	659
q17	1849	1632	1630	1630
q18	8255	7884	7712	7712
q19	1646	1540	1450	1450
q20	2081	1845	1880	1845
q21	5478	5508	5246	5246
q22	1129	1028	1045	1028
Total cold run time: 68805 ms
Total hot run time: 59459 ms

@doris-robot
Copy link

doris-robot commented Apr 14, 2025

TPC-DS: Total hot run time: 197435 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 336c71a9988f85050563383abcf7d3120b9f241e, data reload: false

query1	1270	911	925	911
query2	6260	2109	2051	2051
query3	10818	4137	4325	4137
query4	60915	29277	23607	23607
query5	5164	473	468	468
query6	392	185	196	185
query7	5466	323	317	317
query8	314	230	224	224
query9	8527	2616	2595	2595
query10	474	263	252	252
query11	17899	15122	15653	15122
query12	174	102	110	102
query13	1411	440	436	436
query14	10768	7819	7578	7578
query15	205	174	185	174
query16	7083	458	487	458
query17	1200	586	602	586
query18	1773	348	329	329
query19	222	166	171	166
query20	118	119	111	111
query21	206	103	105	103
query22	4592	4240	4776	4240
query23	34461	34248	33919	33919
query24	6134	2881	2920	2881
query25	532	442	437	437
query26	657	181	181	181
query27	1883	364	369	364
query28	4058	2490	2417	2417
query29	708	469	486	469
query30	243	167	164	164
query31	981	826	876	826
query32	68	61	58	58
query33	447	314	312	312
query34	900	524	515	515
query35	865	759	732	732
query36	1079	968	956	956
query37	122	71	67	67
query38	4111	4081	3997	3997
query39	1548	1665	1474	1474
query40	202	99	99	99
query41	48	47	50	47
query42	109	101	104	101
query43	546	496	497	496
query44	1171	814	833	814
query45	186	167	168	167
query46	1137	714	724	714
query47	2002	1893	1906	1893
query48	485	397	380	380
query49	727	404	393	393
query50	867	430	431	430
query51	7431	7263	7262	7262
query52	111	91	88	88
query53	278	182	184	182
query54	581	477	477	477
query55	85	84	81	81
query56	269	260	256	256
query57	1245	1172	1149	1149
query58	226	220	211	211
query59	3458	2949	3098	2949
query60	283	252	255	252
query61	129	123	111	111
query62	762	675	670	670
query63	216	185	190	185
query64	1388	687	651	651
query65	3252	3188	3196	3188
query66	714	294	292	292
query67	15711	15571	15673	15571
query68	4235	585	571	571
query69	435	263	263	263
query70	1155	1131	1115	1115
query71	340	261	250	250
query72	6341	4049	4035	4035
query73	759	342	364	342
query74	10275	9069	8948	8948
query75	3337	2648	2705	2648
query76	2068	986	1117	986
query77	489	274	277	274
query78	10499	9676	9501	9501
query79	2027	590	588	588
query80	1027	432	430	430
query81	530	239	236	236
query82	185	88	87	87
query83	160	141	157	141
query84	289	77	73	73
query85	959	332	287	287
query86	403	310	280	280
query87	4448	4271	4293	4271
query88	4494	2390	2358	2358
query89	410	291	292	291
query90	1946	185	187	185
query91	180	149	149	149
query92	56	51	51	51
query93	2359	561	575	561
query94	797	292	301	292
query95	361	256	261	256
query96	624	291	280	280
query97	3296	3233	3181	3181
query98	213	200	204	200
query99	1527	1299	1275	1275
Total cold run time: 313891 ms
Total hot run time: 197435 ms

@doris-robot
Copy link

doris-robot commented Apr 14, 2025

ClickBench: Total hot run time: 32.62 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 336c71a9988f85050563383abcf7d3120b9f241e, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.54	0.52	0.51
query6	1.14	0.73	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.52	0.51
query10	0.55	0.55	0.55
query11	0.14	0.10	0.13
query12	0.15	0.12	0.11
query13	0.61	0.60	0.59
query14	2.84	2.83	2.85
query15	0.89	0.84	0.82
query16	0.40	0.38	0.38
query17	1.01	1.06	1.04
query18	0.24	0.21	0.22
query19	1.92	1.85	1.99
query20	0.01	0.01	0.01
query21	15.37	0.58	0.57
query22	2.51	3.36	1.99
query23	16.85	1.06	0.78
query24	3.33	1.10	1.08
query25	0.29	0.26	0.18
query26	0.29	0.14	0.13
query27	0.04	0.05	0.04
query28	10.08	0.50	0.49
query29	12.60	3.18	3.20
query30	0.25	0.06	0.06
query31	2.88	0.38	0.38
query32	3.25	0.46	0.45
query33	2.99	3.05	3.03
query34	17.06	4.47	4.50
query35	4.52	4.52	4.54
query36	0.68	0.47	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.49 s
Total hot run time: 32.62 s

dataroaring
dataroaring approved these changes Apr 22, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 22, 2025
@github-actions
Copy link
Contributor Author

github-actions bot commented Apr 22, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor Author

github-actions bot commented Apr 22, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit 1db091a into branch-3.0 Apr 22, 2025
21 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-49966-branch-3.0 branch April 22, 2025 05:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hello-stephen @doris-robot @dataroaring @morningman @hubgeter