[fix](hudi) upgrade hudi to 0.15.0 #44267

suxiaogang223 · 2024-11-19T08:43:40Z

What problem does this PR solve?

Problem Summary:
upgrade hudi to 0.15.0 to avoid potential class not found problems

Release note

upgrade hudi to 0.15.0.
impl new hudi jni reader based on hudi-hadoop-mr
add session variable hudi_jni_scanner to choose which hudi jni reader to use, "hadoop" means HadoopHudiJniReader, "spark" means old HudiJniReader, default value is "hadoop"
support session variable force_jni_scanner for hudi
add more cases for hudi p2

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-19T08:43:46Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

suxiaogang223 · 2024-11-20T15:38:43Z

run buildall

doris-robot · 2024-11-20T16:18:34Z

TPC-H: Total hot run time: 39897 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 550a2f47ed646fd15758baf12332be335ddfdbf8, data reload: false

------ Round 1 ----------------------------------
q1	17600	7463	7266	7266
q2	2052	167	164	164
q3	10616	1100	1238	1100
q4	10552	761	679	679
q5	7624	2739	2745	2739
q6	248	151	150	150
q7	1000	669	628	628
q8	9246	1861	1911	1861
q9	6553	6406	6377	6377
q10	7054	2308	2288	2288
q11	464	256	278	256
q12	415	216	225	216
q13	17778	3034	3028	3028
q14	238	216	210	210
q15	583	525	528	525
q16	675	583	576	576
q17	983	548	550	548
q18	7307	6633	6716	6633
q19	1332	1040	1066	1040
q20	485	174	182	174
q21	4287	3392	3126	3126
q22	371	315	313	313
Total cold run time: 107463 ms
Total hot run time: 39897 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7224	7282	7268	7268
q2	339	228	223	223
q3	2958	2867	3006	2867
q4	2055	1795	1797	1795
q5	5664	5722	5706	5706
q6	236	141	140	140
q7	2270	1862	1858	1858
q8	3451	3505	3551	3505
q9	8879	8921	8899	8899
q10	3624	3576	3574	3574
q11	598	501	483	483
q12	838	634	612	612
q13	10980	3245	3232	3232
q14	311	270	286	270
q15	582	515	531	515
q16	697	653	652	652
q17	1863	1649	1623	1623
q18	8216	7733	7639	7639
q19	1729	1617	1529	1529
q20	2113	1886	1867	1867
q21	5518	5581	5422	5422
q22	648	587	609	587
Total cold run time: 70793 ms
Total hot run time: 60266 ms

doris-robot · 2024-11-20T16:31:07Z

TPC-DS: Total hot run time: 196370 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 550a2f47ed646fd15758baf12332be335ddfdbf8, data reload: false

query1	1242	947	916	916
query2	6267	2274	2169	2169
query3	10766	4012	3961	3961
query4	67925	29270	23590	23590
query5	4986	457	445	445
query6	415	183	171	171
query7	5482	296	284	284
query8	317	231	231	231
query9	8730	2680	2669	2669
query10	455	252	254	252
query11	17173	15278	15848	15278
query12	154	103	102	102
query13	1434	433	426	426
query14	10904	6968	7003	6968
query15	211	192	185	185
query16	7103	473	445	445
query17	1033	556	558	556
query18	1771	282	284	282
query19	185	159	141	141
query20	115	109	110	109
query21	221	99	99	99
query22	4416	4488	4523	4488
query23	34436	34043	34024	34024
query24	5305	2510	2424	2424
query25	483	379	384	379
query26	638	156	146	146
query27	1692	284	285	284
query28	4334	2434	2418	2418
query29	654	396	400	396
query30	210	145	153	145
query31	1000	783	841	783
query32	65	54	55	54
query33	427	280	283	280
query34	913	512	519	512
query35	828	712	727	712
query36	1067	962	981	962
query37	125	76	78	76
query38	4378	4202	4312	4202
query39	1458	1461	1436	1436
query40	199	98	100	98
query41	48	48	46	46
query42	115	102	99	99
query43	553	515	502	502
query44	1165	809	814	809
query45	188	165	168	165
query46	1127	695	687	687
query47	1998	1871	1889	1871
query48	411	325	326	325
query49	746	406	405	405
query50	860	384	400	384
query51	7306	7036	7117	7036
query52	102	90	87	87
query53	247	174	179	174
query54	513	394	431	394
query55	76	74	77	74
query56	252	227	251	227
query57	1289	1206	1149	1149
query58	221	215	217	215
query59	3378	3142	3376	3142
query60	270	257	250	250
query61	109	109	107	107
query62	816	672	658	658
query63	218	198	189	189
query64	1401	691	659	659
query65	3273	3241	3221	3221
query66	701	306	321	306
query67	16059	15813	15820	15813
query68	3978	561	568	561
query69	424	252	267	252
query70	1186	1162	1150	1150
query71	344	253	255	253
query72	6378	4101	4056	4056
query73	757	359	352	352
query74	10261	9032	9070	9032
query75	3409	2683	2657	2657
query76	1762	1062	1015	1015
query77	488	274	284	274
query78	10603	9461	9479	9461
query79	1153	600	622	600
query80	786	442	441	441
query81	531	243	221	221
query82	240	120	119	119
query83	180	153	149	149
query84	284	72	69	69
query85	836	294	332	294
query86	354	304	294	294
query87	4628	4646	4679	4646
query88	3712	2197	2152	2152
query89	428	306	295	295
query90	2009	188	188	188
query91	131	104	100	100
query92	61	48	54	48
query93	1409	542	544	542
query94	775	294	289	289
query95	347	249	248	248
query96	606	281	278	278
query97	2848	2733	2720	2720
query98	211	196	197	196
query99	1753	1365	1326	1326
Total cold run time: 315831 ms
Total hot run time: 196370 ms

doris-robot · 2024-11-20T16:36:38Z

ClickBench: Total hot run time: 33.32 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 550a2f47ed646fd15758baf12332be335ddfdbf8, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.03
query3	0.23	0.08	0.07
query4	1.62	0.10	0.11
query5	0.44	0.38	0.41
query6	1.15	0.67	0.66
query7	0.02	0.02	0.01
query8	0.05	0.03	0.04
query9	0.57	0.52	0.50
query10	0.56	0.57	0.57
query11	0.15	0.10	0.10
query12	0.13	0.12	0.11
query13	0.62	0.61	0.61
query14	2.70	2.84	2.82
query15	0.89	0.83	0.84
query16	0.39	0.39	0.40
query17	1.06	1.07	1.04
query18	0.20	0.21	0.20
query19	1.96	1.84	2.04
query20	0.01	0.01	0.01
query21	15.38	0.59	0.58
query22	2.59	2.70	2.32
query23	17.04	0.90	0.93
query24	3.19	0.98	1.27
query25	0.19	0.26	0.11
query26	0.50	0.14	0.14
query27	0.04	0.05	0.04
query28	10.69	1.09	1.09
query29	12.56	3.20	3.21
query30	0.25	0.06	0.06
query31	2.87	0.38	0.38
query32	3.28	0.46	0.46
query33	3.00	3.08	3.00
query34	16.97	4.50	4.52
query35	4.52	4.45	4.50
query36	0.64	0.48	0.47
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.15	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 107.05 s
Total hot run time: 33.32 s

suxiaogang223 · 2024-11-21T07:55:46Z

run buildall

github-actions · 2024-11-21T07:58:42Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-11-21T08:32:14Z

TeamCity be ut coverage result:
Function Coverage: 38.04% (9902/26033)
Line Coverage: 29.22% (82846/283514)
Region Coverage: 28.35% (42542/150083)
Branch Coverage: 24.90% (21563/86590)
Coverage Report: http://coverage.selectdb-in.cc/coverage/da2c6d4220c27610dc537441dfcc887ff06a08ed_da2c6d4220c27610dc537441dfcc887ff06a08ed/report/index.html

doris-robot · 2024-11-21T08:35:12Z

TPC-H: Total hot run time: 40093 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit da2c6d4220c27610dc537441dfcc887ff06a08ed, data reload: false

------ Round 1 ----------------------------------
q1	17587	7514	7306	7306
q2	2049	181	173	173
q3	10545	1097	1209	1097
q4	10451	755	834	755
q5	7597	2718	2719	2718
q6	244	150	147	147
q7	998	635	609	609
q8	9452	1887	1903	1887
q9	6609	6473	6520	6473
q10	7055	2329	2324	2324
q11	463	262	253	253
q12	413	229	232	229
q13	19417	3070	3104	3070
q14	247	213	218	213
q15	583	541	516	516
q16	656	598	598	598
q17	983	614	515	515
q18	7305	6832	6711	6711
q19	1329	1034	990	990
q20	471	186	183	183
q21	3987	3222	3008	3008
q22	378	319	318	318
Total cold run time: 108819 ms
Total hot run time: 40093 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7362	7490	7368	7368
q2	340	233	241	233
q3	3057	3136	3041	3041
q4	2183	2043	1950	1950
q5	5825	5831	5900	5831
q6	242	161	153	153
q7	2326	1857	1836	1836
q8	3471	3654	3525	3525
q9	9015	8988	8995	8988
q10	3607	3594	3545	3545
q11	600	523	516	516
q12	831	615	599	599
q13	16572	3371	3194	3194
q14	326	268	288	268
q15	571	536	529	529
q16	688	647	639	639
q17	1858	1634	1639	1634
q18	8514	7678	7810	7678
q19	2404	1571	1508	1508
q20	2131	1876	1892	1876
q21	5593	5538	5365	5365
q22	621	551	582	551
Total cold run time: 78137 ms
Total hot run time: 60827 ms

doris-robot · 2024-11-21T08:47:55Z

TPC-DS: Total hot run time: 197412 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit da2c6d4220c27610dc537441dfcc887ff06a08ed, data reload: false

query1	1271	944	928	928
query2	6247	2108	2047	2047
query3	10775	4041	3970	3970
query4	67474	29435	23721	23721
query5	4933	463	461	461
query6	421	192	180	180
query7	5542	307	290	290
query8	316	229	229	229
query9	8625	2686	2657	2657
query10	467	272	288	272
query11	17288	15341	15929	15341
query12	161	110	109	109
query13	1477	414	430	414
query14	10837	7444	7550	7444
query15	226	198	197	197
query16	7099	468	488	468
query17	1034	563	574	563
query18	1833	307	312	307
query19	213	160	149	149
query20	126	108	114	108
query21	205	108	114	108
query22	4710	4492	4624	4492
query23	34812	34207	34442	34207
query24	5490	2499	2583	2499
query25	494	400	415	400
query26	682	153	157	153
query27	1902	287	296	287
query28	4332	2457	2444	2444
query29	685	428	427	427
query30	223	152	150	150
query31	997	817	840	817
query32	65	59	64	59
query33	405	296	276	276
query34	928	528	511	511
query35	893	739	738	738
query36	1123	988	997	988
query37	117	69	77	69
query38	4458	4457	4422	4422
query39	1514	1483	1451	1451
query40	202	100	105	100
query41	48	43	45	43
query42	113	99	102	99
query43	541	516	501	501
query44	1206	855	832	832
query45	189	178	167	167
query46	1174	717	701	701
query47	2027	1915	1950	1915
query48	437	340	334	334
query49	748	419	422	419
query50	844	397	396	396
query51	7406	7229	7028	7028
query52	103	89	92	89
query53	268	181	178	178
query54	539	410	405	405
query55	81	78	89	78
query56	263	260	264	260
query57	1337	1200	1138	1138
query58	219	208	222	208
query59	3401	3064	2961	2961
query60	266	249	239	239
query61	113	108	110	108
query62	802	669	664	664
query63	217	189	185	185
query64	1370	681	724	681
query65	3309	3242	3234	3234
query66	720	287	308	287
query67	16004	15678	15764	15678
query68	4018	585	569	569
query69	422	255	262	255
query70	1270	1132	1097	1097
query71	363	243	257	243
query72	6358	4040	4046	4040
query73	758	359	364	359
query74	10389	9162	8903	8903
query75	3430	2728	2688	2688
query76	1880	996	1156	996
query77	484	275	279	275
query78	10511	9476	9380	9380
query79	2152	611	605	605
query80	1390	427	440	427
query81	531	227	225	225
query82	1277	187	117	117
query83	269	163	149	149
query84	285	79	72	72
query85	1014	316	298	298
query86	420	312	303	303
query87	4781	4721	4681	4681
query88	3563	2267	2309	2267
query89	412	303	297	297
query90	1956	191	189	189
query91	135	106	101	101
query92	68	49	51	49
query93	2695	543	548	543
query94	826	301	286	286
query95	360	253	250	250
query96	634	285	276	276
query97	2869	2698	2708	2698
query98	226	192	198	192
query99	1601	1321	1301	1301
Total cold run time: 321694 ms
Total hot run time: 197412 ms

doris-robot · 2024-11-21T08:53:25Z

ClickBench: Total hot run time: 32.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit da2c6d4220c27610dc537441dfcc887ff06a08ed, data reload: false

query1	0.03	0.04	0.03
query2	0.06	0.04	0.03
query3	0.24	0.07	0.07
query4	1.63	0.10	0.11
query5	0.41	0.41	0.41
query6	1.14	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.02	0.03
query9	0.58	0.51	0.51
query10	0.56	0.56	0.56
query11	0.15	0.10	0.11
query12	0.13	0.10	0.11
query13	0.61	0.60	0.60
query14	2.77	2.83	2.75
query15	0.91	0.82	0.83
query16	0.37	0.38	0.40
query17	1.07	1.05	1.06
query18	0.22	0.21	0.20
query19	1.96	1.78	1.96
query20	0.02	0.01	0.01
query21	15.37	0.59	0.58
query22	2.49	2.57	2.21
query23	17.11	0.93	0.75
query24	2.63	0.78	1.17
query25	0.27	0.06	0.06
query26	0.40	0.13	0.14
query27	0.04	0.04	0.05
query28	11.35	1.09	1.07
query29	12.52	3.24	3.24
query30	0.25	0.06	0.06
query31	2.87	0.38	0.37
query32	3.31	0.46	0.46
query33	2.99	3.03	3.06
query34	16.63	4.50	4.43
query35	4.56	4.46	4.55
query36	0.68	0.49	0.48
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.12	0.12
query41	0.09	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.9 s
Total hot run time: 32.65 s

github-actions · 2024-11-21T09:22:54Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-11-21T09:33:25Z

clang-tidy review says "All clean, LGTM! 👍"

suxiaogang223 · 2024-11-22T02:02:51Z

run buildall

suxiaogang223 · 2024-11-22T02:07:38Z

run buildall

doris-robot · 2024-11-22T03:41:14Z

TPC-H: Total hot run time: 40408 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01413781a887846e4a866acc572e23fc4a456004, data reload: false

------ Round 1 ----------------------------------
q1	17633	7617	7298	7298
q2	2059	174	186	174
q3	10706	1116	1162	1116
q4	10447	818	743	743
q5	7605	2864	2741	2741
q6	244	145	144	144
q7	992	637	624	624
q8	9251	1920	1959	1920
q9	6639	6464	6473	6464
q10	7053	2331	2330	2330
q11	472	259	260	259
q12	436	219	211	211
q13	17790	3064	3052	3052
q14	241	208	211	208
q15	570	523	522	522
q16	674	587	589	587
q17	1002	583	599	583
q18	7401	6738	6726	6726
q19	1363	1058	975	975
q20	469	182	182	182
q21	3966	3233	3248	3233
q22	383	321	316	316
Total cold run time: 107396 ms
Total hot run time: 40408 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7313	7817	7267	7267
q2	326	228	231	228
q3	2920	2947	2994	2947
q4	2066	1874	1844	1844
q5	5695	5719	5740	5719
q6	231	140	141	140
q7	2265	1863	1841	1841
q8	3449	3609	3550	3550
q9	8957	9000	9054	9000
q10	3618	3564	3593	3564
q11	598	523	499	499
q12	827	614	598	598
q13	10428	3243	3265	3243
q14	307	263	292	263
q15	592	509	522	509
q16	721	661	662	661
q17	1887	1654	1613	1613
q18	8537	7884	7725	7725
q19	1809	1569	1612	1569
q20	2106	1902	1910	1902
q21	5719	5518	5379	5379
q22	658	565	549	549
Total cold run time: 71029 ms
Total hot run time: 60610 ms

doris-robot · 2024-11-22T03:49:42Z

TeamCity be ut coverage result:
Function Coverage: 38.31% (9977/26045)
Line Coverage: 29.41% (83446/283754)
Region Coverage: 28.56% (42922/150302)
Branch Coverage: 25.15% (21812/86718)
Coverage Report: http://coverage.selectdb-in.cc/coverage/01413781a887846e4a866acc572e23fc4a456004_01413781a887846e4a866acc572e23fc4a456004/report/index.html

github-actions · 2024-11-22T06:52:53Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-11-22T09:53:53Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-11-25T07:50:19Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-11-25T11:18:49Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-11-25T11:47:45Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-11-25T15:50:17Z

clang-tidy review says "All clean, LGTM! 👍"

gensrc/thrift/PlanNodes.thrift

...extensions/hadoop-hudi-scanner/src/main/java/org/apache/doris/hudi/HadoopHudiJniScanner.java

suxiaogang223 · 2024-12-02T12:10:45Z

run buildall

doris-robot · 2024-12-02T12:49:00Z

TeamCity be ut coverage result:
Function Coverage: 38.39% (9978/25989)
Line Coverage: 29.46% (83688/284059)
Region Coverage: 28.58% (43049/150617)
Branch Coverage: 25.21% (21888/86834)
Coverage Report: http://coverage.selectdb-in.cc/coverage/093f49e8380bd1193739fa32d9cf6332f1e61cec_093f49e8380bd1193739fa32d9cf6332f1e61cec/report/index.html

doris-robot · 2024-12-02T13:00:48Z

TPC-H: Total hot run time: 40078 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 093f49e8380bd1193739fa32d9cf6332f1e61cec, data reload: false

------ Round 1 ----------------------------------
q1	17912	7785	7342	7342
q2	2044	183	188	183
q3	10818	1093	1195	1093
q4	10468	738	781	738
q5	7606	2721	2742	2721
q6	241	149	147	147
q7	1020	617	614	614
q8	9241	1850	1920	1850
q9	6658	6415	6528	6415
q10	7040	2294	2337	2294
q11	477	257	270	257
q12	431	233	235	233
q13	17770	3042	3068	3042
q14	238	212	209	209
q15	600	540	518	518
q16	664	588	600	588
q17	982	559	494	494
q18	7276	6752	6653	6653
q19	1344	909	964	909
q20	468	184	181	181
q21	4027	3321	3286	3286
q22	385	311	324	311
Total cold run time: 107710 ms
Total hot run time: 40078 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7395	7296	7289	7289
q2	324	239	225	225
q3	2925	2933	2961	2933
q4	2167	1838	1861	1838
q5	5653	5678	5677	5677
q6	227	144	139	139
q7	2262	1818	1828	1818
q8	3413	3540	3515	3515
q9	8873	8940	8974	8940
q10	3618	3547	3626	3547
q11	586	513	518	513
q12	859	595	645	595
q13	12184	3299	3259	3259
q14	300	269	295	269
q15	577	519	538	519
q16	709	670	669	669
q17	1881	1614	1615	1614
q18	8257	7790	7572	7572
q19	1688	1533	1622	1533
q20	2123	1901	1865	1865
q21	5854	5520	5597	5520
q22	639	594	567	567
Total cold run time: 72514 ms
Total hot run time: 60416 ms

doris-robot · 2024-12-02T13:13:29Z

TPC-DS: Total hot run time: 196867 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 093f49e8380bd1193739fa32d9cf6332f1e61cec, data reload: false

query1	1252	991	962	962
query2	6264	2015	2083	2015
query3	11081	4573	4414	4414
query4	66809	29165	23490	23490
query5	4998	465	453	453
query6	419	198	182	182
query7	5517	310	316	310
query8	333	243	248	243
query9	8840	2685	2676	2676
query10	442	245	250	245
query11	17176	15209	15851	15209
query12	156	103	100	100
query13	1429	418	406	406
query14	9831	6770	7956	6770
query15	225	188	193	188
query16	7127	507	493	493
query17	1065	589	607	589
query18	1694	311	324	311
query19	216	162	166	162
query20	119	114	111	111
query21	216	109	103	103
query22	4667	4595	4577	4577
query23	34721	34215	34474	34215
query24	5560	2486	2590	2486
query25	497	386	375	375
query26	666	148	154	148
query27	1872	282	288	282
query28	4283	2492	2446	2446
query29	711	413	411	411
query30	205	149	148	148
query31	1006	813	877	813
query32	64	54	57	54
query33	431	284	289	284
query34	914	522	529	522
query35	888	761	780	761
query36	1109	976	993	976
query37	131	70	71	70
query38	4596	4503	4485	4485
query39	1518	1448	1500	1448
query40	207	97	102	97
query41	44	41	44	41
query42	105	103	103	103
query43	538	483	502	483
query44	1211	825	823	823
query45	198	167	169	167
query46	1172	719	704	704
query47	2033	2016	1925	1925
query48	406	306	333	306
query49	722	395	383	383
query50	878	407	408	407
query51	7455	7174	7135	7135
query52	98	86	86	86
query53	246	182	177	177
query54	518	392	408	392
query55	86	73	80	73
query56	266	251	235	235
query57	1257	1135	1104	1104
query58	214	207	213	207
query59	3108	3023	2951	2951
query60	257	238	232	232
query61	102	110	103	103
query62	828	672	666	666
query63	213	215	198	198
query64	1334	658	629	629
query65	3355	3184	3229	3184
query66	699	302	305	302
query67	16074	15699	15776	15699
query68	3860	565	551	551
query69	430	255	251	251
query70	1204	1072	1144	1072
query71	372	250	248	248
query72	6366	4151	4054	4054
query73	772	365	361	361
query74	10272	9010	8997	8997
query75	3372	2802	2726	2726
query76	2010	1043	1137	1043
query77	532	272	267	267
query78	10389	9506	9472	9472
query79	2137	604	602	602
query80	1355	415	424	415
query81	516	229	241	229
query82	1290	120	128	120
query83	200	140	137	137
query84	282	72	73	72
query85	1008	291	293	291
query86	417	296	297	296
query87	4686	4571	4546	4546
query88	3828	2229	2195	2195
query89	417	309	291	291
query90	1936	189	188	188
query91	131	100	102	100
query92	68	48	58	48
query93	2753	548	550	548
query94	841	283	331	283
query95	356	248	249	248
query96	625	278	279	278
query97	2871	2688	2679	2679
query98	213	197	198	197
query99	1613	1317	1349	1317
Total cold run time: 319794 ms
Total hot run time: 196867 ms

doris-robot · 2024-12-02T13:19:13Z

ClickBench: Total hot run time: 33.71 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 093f49e8380bd1193739fa32d9cf6332f1e61cec, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.05	0.05
query3	0.23	0.06	0.05
query4	1.65	0.09	0.08
query5	0.41	0.39	0.41
query6	1.16	0.67	0.65
query7	0.02	0.02	0.02
query8	0.05	0.05	0.04
query9	0.54	0.51	0.49
query10	0.55	0.55	0.56
query11	0.17	0.12	0.12
query12	0.16	0.13	0.12
query13	0.62	0.60	0.59
query14	2.70	2.85	2.76
query15	0.93	0.84	0.84
query16	0.38	0.38	0.38
query17	1.08	1.06	1.08
query18	0.18	0.19	0.18
query19	1.98	1.77	2.03
query20	0.02	0.01	0.01
query21	15.36	0.68	0.68
query22	3.98	7.52	2.19
query23	18.28	1.41	1.33
query24	2.06	0.22	0.24
query25	0.15	0.09	0.08
query26	0.26	0.18	0.18
query27	0.09	0.08	0.08
query28	13.28	1.16	1.16
query29	12.62	3.43	3.41
query30	0.25	0.07	0.06
query31	2.86	0.42	0.41
query32	3.23	0.49	0.48
query33	3.03	3.03	3.14
query34	16.73	4.53	4.59
query35	4.60	4.59	4.61
query36	0.69	0.48	0.48
query37	0.19	0.16	0.17
query38	0.16	0.15	0.15
query39	0.06	0.04	0.04
query40	0.17	0.14	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.05
Total cold run time: 111.2 s
Total hot run time: 33.71 s

github-actions · 2024-12-04T00:59:04Z

PR approved by at least one committer and no changes requested.

### What problem does this PR solve? 1. upgrade hudi to 0.15.0. 2. impl new hudi jni reader based on hudi-hadoop-mr 3. add session variable `hudi_jni_scanner` to choose which hudi jni reader to use, "hadoop" means HadoopHudiJniReader, "spark" means old HudiJniReader, default value is "hadoop" 4. support session variable `force_jni_scanner` for hudi 5. add more cases for hudi p2 ### Release note [opt](hudi) upgrade hudi to 0.15 and support hadoop jni reader

1. upgrade hudi to 0.15.0. 2. impl new hudi jni reader based on hudi-hadoop-mr 3. add session variable `hudi_jni_scanner` to choose which hudi jni reader to use, "hadoop" means HadoopHudiJniReader, "spark" means old HudiJniReader, default value is "hadoop" 4. support session variable `force_jni_scanner` for hudi 5. add more cases for hudi p2 [opt](hudi) upgrade hudi to 0.15 and support hadoop jni reader

cherry-pick (#44267)(#44995)

cherry-pick pr: #44267 #44995

### What problem does this PR solve? Related PR: #44267 Problem Summary: If we set hudi_jni_scanner to an incorrect value, jni_connector will be null, causing a core dump. So we set hadoop as default hudi jni scanner.

### What problem does this PR solve? Related PR: apache#44267 Problem Summary: If we set hudi_jni_scanner to an incorrect value, jni_connector will be null, causing a core dump. So we set hadoop as default hudi jni scanner.

suxiaogang223 requested review from CalvinKirs and morningman as code owners November 19, 2024 08:43

suxiaogang223 force-pushed the update_hudi_0.15.0 branch from 0141378 to 9147fd3 Compare November 22, 2024 06:48

suxiaogang223 force-pushed the update_hudi_0.15.0 branch from 081336b to d0c8116 Compare November 25, 2024 11:42

suxiaogang223 marked this pull request as draft November 26, 2024 14:42

suxiaogang223 force-pushed the update_hudi_0.15.0 branch from 04a6609 to 35a4152 Compare November 26, 2024 14:46

wuwenchi reviewed Nov 27, 2024

View reviewed changes

gensrc/thrift/PlanNodes.thrift Outdated Show resolved Hide resolved

...extensions/hadoop-hudi-scanner/src/main/java/org/apache/doris/hudi/HadoopHudiJniScanner.java Outdated Show resolved Hide resolved

wuwenchi approved these changes Dec 3, 2024

View reviewed changes

morningman approved these changes Dec 4, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 4, 2024

morningman merged commit 142c693 into apache:master Dec 4, 2024
15 of 16 checks passed

github-actions bot mentioned this pull request Dec 4, 2024

branch-3.0: [fix](hudi) upgrade hudi to 0.15.0 #44267 #44961

Closed

github-actions bot added the dev/2.1.x-conflict label Dec 4, 2024

suxiaogang223 deleted the update_hudi_0.15.0 branch December 5, 2024 05:59

suxiaogang223 mentioned this pull request Dec 5, 2024

branch-2.1: [fix](hudi) upgrade hudi to 0.15.0 (#44267)(#44995) #45041

Merged

16 tasks

morningman pushed a commit that referenced this pull request Dec 5, 2024

branch-3.0: [fix](hudi) upgrade hudi to 0.15.0 (#44267)(#44995) (#45054)

d6482e1

cherry-pick (#44267)(#44995)

morningman mentioned this pull request Dec 5, 2024

branch-3.0: [fix](hudi) upgrade hudi to 0.15.0 (#44267)(#44995) #45054

Merged

morningman added dev/3.0.4-merged and removed dev/3.0.x labels Dec 5, 2024

morningman pushed a commit that referenced this pull request Dec 6, 2024

branch-2.1: [fix](hudi) upgrade hudi to 0.15.0 (#44267)(#44995) (#45041)

53291bd

cherry-pick pr: #44267 #44995

morningman added dev/2.1.8-merged and removed dev/2.1.x dev/2.1.x-conflict labels Dec 6, 2024

gavinchou mentioned this pull request Feb 18, 2025

Release Notes 3.0.4 #48013

Closed

suxiaogang223 mentioned this pull request Mar 4, 2025

[fix](hudi) Set Hadoop Hudi Jni reader as default #48602

Merged

16 tasks

suxiaogang223 mentioned this pull request Mar 7, 2025

[fix](hudi-jni) kill jol process #48803

Closed

16 tasks

[fix](hudi) upgrade hudi to 0.15.0 #44267

[fix](hudi) upgrade hudi to 0.15.0 #44267

Uh oh!

Conversation

suxiaogang223 commented Nov 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Nov 19, 2024

Uh oh!

suxiaogang223 commented Nov 20, 2024

Uh oh!

doris-robot commented Nov 20, 2024

Uh oh!

doris-robot commented Nov 20, 2024

Uh oh!

doris-robot commented Nov 20, 2024

Uh oh!

suxiaogang223 commented Nov 21, 2024

Uh oh!

github-actions bot commented Nov 21, 2024

Uh oh!

doris-robot commented Nov 21, 2024

Uh oh!

doris-robot commented Nov 21, 2024

Uh oh!

doris-robot commented Nov 21, 2024

Uh oh!

doris-robot commented Nov 21, 2024

Uh oh!

github-actions bot commented Nov 21, 2024

Uh oh!

github-actions bot commented Nov 21, 2024

Uh oh!

suxiaogang223 commented Nov 22, 2024

Uh oh!

suxiaogang223 commented Nov 22, 2024

Uh oh!

doris-robot commented Nov 22, 2024

Uh oh!

doris-robot commented Nov 22, 2024

Uh oh!

github-actions bot commented Nov 22, 2024

Uh oh!

github-actions bot commented Nov 22, 2024

Uh oh!

github-actions bot commented Nov 25, 2024

Uh oh!

github-actions bot commented Nov 25, 2024

Uh oh!

github-actions bot commented Nov 25, 2024

Uh oh!

github-actions bot commented Nov 25, 2024

Uh oh!

Uh oh!

Uh oh!

suxiaogang223 commented Dec 2, 2024

Uh oh!

doris-robot commented Dec 2, 2024

Uh oh!

doris-robot commented Dec 2, 2024

Uh oh!

doris-robot commented Dec 2, 2024

Uh oh!

doris-robot commented Dec 2, 2024

Uh oh!

github-actions bot commented Dec 4, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

suxiaogang223 commented Nov 19, 2024 •

edited

Loading