[deps](jindofs) add jindofs for 2.1 (#49259)(#49371)(#49374)(#49502)(#49503)(#49886)(#49903)(#49963) #50283

morningman · 2025-04-22T05:03:43Z

bp (#49259)(#49371)(#49374)(#49502)(#49503)(#49886)(#49903)(#49963)

…he#49374)(apache#49502)(apache#49503)(apache#49886)(apache#49903) (apache#49908) bp (apache#49259)(apache#49371)(apache#49374)(apache#49502)(apache#49503)(apache#49886)(apache#49903)

…#49374) Followup apache#49371 Problem Summary: The `jindofs-core` and `jindofs-sdk` jars will be installed in `fe/lib` and `be/lib/java_extensions/preload-extensions`. So that user don't need to install these jars to access aliyun oss-hdfs manually.

### What problem does this PR solve? Followup apache#49502 1. add jindofs ubuntu jar in build.sh, for ubuntu 22 OS. 2. set `fs.oss.impl` after check if it is oss-hdfs, to avoid depending on both `com.aliyun.jindodata.oss.JindoOssFileSystem` and `org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem`

### What problem does this PR solve? Followup apache#49886 Use regexp to match the different version of jindofs jars. So that we don't need to modify `build.sh` again once the version of jindofs is changed

cp: cannot stat '/home/zcp/repo_center/doris_master/doris/thirdparty/installed/jindofs_libs/jindo-core-linux-el7-aarch64-[0-9]*.jar': No such file or directory

hello-stephen · 2025-04-22T05:03:56Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2025-04-22T05:04:07Z

run buildall

hello-stephen · 2025-04-22T07:06:30Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	40.21% (10555/26248)
Line Coverage	30.98% (89297/288213)
Region Coverage	30.11% (46060/152965)
Branch Coverage	26.62% (23544/88458)

morningman · 2025-04-22T20:04:43Z

run buildall

doris-robot · 2025-04-22T20:43:15Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	40.22% (10558/26248)
Line Coverage	31.00% (89347/288214)
Region Coverage	30.12% (46074/152965)
Branch Coverage	26.63% (23552/88458)

doris-robot · 2025-04-22T20:44:02Z

TPC-H: Total hot run time: 39928 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4aba8a93d8c600a93493ad9ddaf67b99baadec92, data reload: false

------ Round 1 ----------------------------------
q1	17619	6734	6577	6577
q2	2068	169	171	169
q3	10649	1081	1195	1081
q4	10560	746	722	722
q5	7749	2799	2783	2783
q6	219	138	133	133
q7	982	633	606	606
q8	9350	1970	1998	1970
q9	6511	6389	6401	6389
q10	7106	2281	2345	2281
q11	469	273	267	267
q12	397	212	216	212
q13	17809	2969	2976	2969
q14	235	208	210	208
q15	493	451	457	451
q16	669	595	575	575
q17	964	584	581	581
q18	7092	6495	6596	6495
q19	1433	1094	1097	1094
q20	462	204	208	204
q21	4283	3219	3194	3194
q22	1080	967	976	967
Total cold run time: 108199 ms
Total hot run time: 39928 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6598	6538	6537	6537
q2	328	234	225	225
q3	2885	2761	2891	2761
q4	2037	1809	1748	1748
q5	5715	5746	5731	5731
q6	205	130	126	126
q7	2186	1790	1844	1790
q8	3308	3507	3508	3507
q9	8973	8744	8791	8744
q10	3571	3534	3509	3509
q11	585	498	502	498
q12	843	622	639	622
q13	8839	3145	3193	3145
q14	318	283	264	264
q15	511	467	476	467
q16	675	654	659	654
q17	1826	1635	1624	1624
q18	8141	7799	7708	7708
q19	1685	1530	1549	1530
q20	2039	1832	1808	1808
q21	5441	5317	5385	5317
q22	1097	1034	1004	1004
Total cold run time: 67806 ms
Total hot run time: 59319 ms

doris-robot · 2025-04-22T20:56:32Z

TPC-DS: Total hot run time: 196065 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4aba8a93d8c600a93493ad9ddaf67b99baadec92, data reload: false

query1	1272	912	888	888
query2	6238	1968	1917	1917
query3	10835	4138	4290	4138
query4	61288	28947	23377	23377
query5	5146	470	445	445
query6	380	178	183	178
query7	5471	315	303	303
query8	309	226	223	223
query9	8241	2626	2606	2606
query10	458	278	254	254
query11	17640	15226	15796	15226
query12	166	109	104	104
query13	1459	450	441	441
query14	10679	7392	6801	6801
query15	212	190	183	183
query16	7062	500	500	500
query17	1124	612	643	612
query18	1824	340	331	331
query19	207	166	169	166
query20	120	111	108	108
query21	207	108	106	106
query22	4686	4393	4757	4393
query23	34277	34020	33817	33817
query24	6197	2964	2876	2876
query25	550	428	421	421
query26	652	169	180	169
query27	1769	363	382	363
query28	4649	2470	2445	2445
query29	716	483	444	444
query30	255	163	162	162
query31	983	850	813	813
query32	66	59	58	58
query33	474	294	295	294
query34	916	515	533	515
query35	884	722	731	722
query36	1092	984	959	959
query37	116	68	64	64
query38	4043	4000	3939	3939
query39	1515	1499	1477	1477
query40	205	100	109	100
query41	46	48	46	46
query42	112	101	102	101
query43	522	484	495	484
query44	1197	815	829	815
query45	185	174	165	165
query46	1172	729	742	729
query47	1971	1905	1946	1905
query48	469	385	397	385
query49	705	396	392	392
query50	839	438	429	429
query51	7421	7160	7255	7160
query52	102	89	88	88
query53	265	185	191	185
query54	566	483	472	472
query55	79	77	76	76
query56	262	260	252	252
query57	1291	1203	1135	1135
query58	213	207	205	205
query59	3166	3018	2915	2915
query60	284	256	250	250
query61	111	123	104	104
query62	759	678	673	673
query63	213	197	193	193
query64	1400	659	666	659
query65	3284	3198	3189	3189
query66	691	300	296	296
query67	15964	15557	15452	15452
query68	4191	579	575	575
query69	437	271	261	261
query70	1201	1140	1091	1091
query71	354	264	263	263
query72	6318	3906	3981	3906
query73	759	342	357	342
query74	10170	9302	9214	9214
query75	3349	2634	2630	2630
query76	1875	1043	963	963
query77	474	274	267	267
query78	10552	9563	9599	9563
query79	1350	597	603	597
query80	956	429	437	429
query81	539	243	236	236
query82	188	87	87	87
query83	173	149	145	145
query84	278	83	87	83
query85	947	292	296	292
query86	373	298	302	298
query87	4435	4193	4323	4193
query88	3871	2387	2355	2355
query89	409	290	296	290
query90	2014	186	185	185
query91	189	153	144	144
query92	64	48	49	48
query93	2219	555	556	555
query94	788	287	293	287
query95	370	272	264	264
query96	623	274	284	274
query97	3319	3077	3252	3077
query98	215	203	195	195
query99	1570	1294	1263	1263
Total cold run time: 312242 ms
Total hot run time: 196065 ms

doris-robot · 2025-04-22T21:02:01Z

ClickBench: Total hot run time: 32.39 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4aba8a93d8c600a93493ad9ddaf67b99baadec92, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.04	0.03
query3	0.23	0.06	0.06
query4	1.62	0.10	0.10
query5	0.53	0.50	0.52
query6	1.13	0.72	0.74
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.57	0.51	0.49
query10	0.55	0.55	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.62	0.59	0.60
query14	2.88	2.74	2.75
query15	0.90	0.83	0.82
query16	0.37	0.37	0.40
query17	1.07	1.04	1.08
query18	0.24	0.23	0.22
query19	1.88	1.84	2.03
query20	0.02	0.01	0.01
query21	15.35	0.56	0.57
query22	2.88	2.15	2.14
query23	16.97	0.93	0.92
query24	3.12	1.67	0.91
query25	0.29	0.18	0.04
query26	0.42	0.13	0.14
query27	0.05	0.03	0.04
query28	10.03	0.53	0.50
query29	12.55	3.22	3.22
query30	0.25	0.07	0.05
query31	2.86	0.39	0.38
query32	3.24	0.46	0.45
query33	2.97	2.98	2.97
query34	17.44	4.50	4.46
query35	4.52	4.50	4.53
query36	0.66	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 107.12 s
Total hot run time: 32.39 s

dataroaring

LGTM

morningman and others added 5 commits April 21, 2025 21:59

[deps](jindofs) add jindofs for 2.1 (apache#49259)(apache#49371)(apac…

79bb9b1

…he#49374)(apache#49502)(apache#49503)(apache#49886)(apache#49903) (apache#49908) bp (apache#49259)(apache#49371)(apache#49374)(apache#49502)(apache#49503)(apache#49886)(apache#49903)

[deps](jindofs) build.sh support any version of jindofs (apache#49903)

b337e98

### What problem does this PR solve? Followup apache#49886 Use regexp to match the different version of jindofs jars. So that we don't need to modify `build.sh` again once the version of jindofs is changed

[fix](compile) fix typo (apache#49963)

275c44d

cp: cannot stat '/home/zcp/repo_center/doris_master/doris/thirdparty/installed/jindofs_libs/jindo-core-linux-el7-aarch64-[0-9]*.jar': No such file or directory

morningman requested a review from dataroaring as a code owner April 22, 2025 05:03

fix cmake

4aba8a9

dataroaring approved these changes Apr 23, 2025

View reviewed changes

dataroaring merged commit 8aff230 into apache:branch-3.0 Apr 23, 2025
21 of 23 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[deps](jindofs) add jindofs for 2.1 (#49259)(#49371)(#49374)(#49502)(#49503)(#49886)(#49903)(#49963) #50283

[deps](jindofs) add jindofs for 2.1 (#49259)(#49371)(#49374)(#49502)(#49503)(#49886)(#49903)(#49963) #50283

Uh oh!

morningman commented Apr 22, 2025

Uh oh!

hello-stephen commented Apr 22, 2025

Uh oh!

morningman commented Apr 22, 2025

Uh oh!

hello-stephen commented Apr 22, 2025

Uh oh!

morningman commented Apr 22, 2025

Uh oh!

doris-robot commented Apr 22, 2025

Uh oh!

doris-robot commented Apr 22, 2025

Uh oh!

doris-robot commented Apr 22, 2025

Uh oh!

doris-robot commented Apr 22, 2025

Uh oh!

dataroaring left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[deps](jindofs) add jindofs for 2.1 (#49259)(#49371)(#49374)(#49502)(#49503)(#49886)(#49903)(#49963) #50283

[deps](jindofs) add jindofs for 2.1 (#49259)(#49371)(#49374)(#49502)(#49503)(#49886)(#49903)(#49963) #50283

Uh oh!

Conversation

morningman commented Apr 22, 2025

Uh oh!

hello-stephen commented Apr 22, 2025

Uh oh!

morningman commented Apr 22, 2025

Uh oh!

hello-stephen commented Apr 22, 2025

BE UT Coverage Report

Uh oh!

morningman commented Apr 22, 2025

Uh oh!

doris-robot commented Apr 22, 2025

BE UT Coverage Report

Uh oh!

doris-robot commented Apr 22, 2025

Uh oh!

doris-robot commented Apr 22, 2025

Uh oh!

doris-robot commented Apr 22, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants