Skip to content

[fix](jindofs) add jindofs ubuntu jar in build.sh #49503

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 3 commits into from
Mar 27, 2025
Merged

[fix](jindofs) add jindofs ubuntu jar in build.sh #49503

morningman merged 3 commits into from
Mar 27, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Mar 26, 2025 •
edited
Loading

What problem does this PR solve?

Followup #49502

  1. add jindofs ubuntu jar in build.sh, for ubuntu 22 OS.
  2. set fs.oss.impl after check if it is oss-hdfs, to avoid depending on both com.aliyun.jindodata.oss.JindoOssFileSystem and org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@morningman morningman requested a review from CalvinKirs as a code owner March 26, 2025 04:26
@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 26, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

morningman commented Mar 26, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

TPC-H: Total hot run time: 34331 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 27256d3f49982211a73d122a0927bbeb43c5615c, data reload: false

------ Round 1 ----------------------------------
q1	26255	5133	5023	5023
q2	2082	294	174	174
q3	10396	1258	689	689
q4	10224	996	542	542
q5	7495	2380	2387	2380
q6	188	169	138	138
q7	924	724	629	629
q8	9321	1370	1095	1095
q9	7015	5109	5144	5109
q10	6815	2331	1886	1886
q11	468	269	253	253
q12	351	353	220	220
q13	17765	3705	3096	3096
q14	225	233	212	212
q15	537	488	481	481
q16	630	624	599	599
q17	584	883	347	347
q18	7590	7278	7257	7257
q19	1229	948	574	574
q20	317	341	210	210
q21	4065	2620	2444	2444
q22	1050	1040	973	973
Total cold run time: 115526 ms
Total hot run time: 34331 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5111	5093	5081	5081
q2	258	324	223	223
q3	2156	2662	2315	2315
q4	1419	1776	1391	1391
q5	4562	4478	4408	4408
q6	221	165	121	121
q7	1932	1896	1790	1790
q8	2552	2485	2438	2438
q9	7163	7126	7155	7126
q10	2978	3170	2772	2772
q11	573	502	480	480
q12	661	769	657	657
q13	3497	3916	3311	3311
q14	296	301	277	277
q15	519	475	490	475
q16	639	687	646	646
q17	1159	1564	1391	1391
q18	7744	7573	7407	7407
q19	834	816	854	816
q20	2017	2013	1911	1911
q21	5150	4779	4571	4571
q22	1042	1012	958	958
Total cold run time: 52483 ms
Total hot run time: 50565 ms

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

TPC-DS: Total hot run time: 186266 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 27256d3f49982211a73d122a0927bbeb43c5615c, data reload: false

query1	1011	486	489	486
query2	6542	1900	1899	1899
query3	6808	217	214	214
query4	25948	23905	23028	23028
query5	4317	669	498	498
query6	312	203	189	189
query7	4610	494	280	280
query8	307	259	242	242
query9	8645	2547	2545	2545
query10	481	315	251	251
query11	15304	15317	14989	14989
query12	155	110	111	110
query13	1675	547	418	418
query14	8770	6195	6083	6083
query15	213	189	168	168
query16	7272	638	451	451
query17	1221	740	559	559
query18	1978	401	324	324
query19	191	194	158	158
query20	122	122	118	118
query21	217	124	104	104
query22	4195	4324	4074	4074
query23	33995	33200	33119	33119
query24	8463	2369	2366	2366
query25	531	456	404	404
query26	1239	278	147	147
query27	2729	497	315	315
query28	4319	2390	2402	2390
query29	766	559	427	427
query30	282	222	189	189
query31	955	874	772	772
query32	74	70	62	62
query33	556	379	301	301
query34	767	881	502	502
query35	771	820	737	737
query36	967	992	887	887
query37	115	110	78	78
query38	4328	4171	4238	4171
query39	1450	1413	1423	1413
query40	209	114	104	104
query41	57	56	54	54
query42	119	104	106	104
query43	498	501	487	487
query44	1287	809	812	809
query45	177	179	168	168
query46	846	1056	642	642
query47	1839	1843	1811	1811
query48	373	403	296	296
query49	776	528	420	420
query50	691	729	418	418
query51	4149	4202	4147	4147
query52	108	105	103	103
query53	226	248	179	179
query54	481	498	406	406
query55	82	81	82	81
query56	269	272	262	262
query57	1166	1168	1098	1098
query58	246	238	252	238
query59	2547	2590	2589	2589
query60	290	282	261	261
query61	132	172	130	130
query62	787	715	674	674
query63	225	184	178	178
query64	4408	1025	691	691
query65	4355	4280	4228	4228
query66	1158	411	298	298
query67	15878	15726	15617	15617
query68	8671	889	506	506
query69	477	299	260	260
query70	1235	1081	1091	1081
query71	449	292	268	268
query72	5788	4663	4818	4663
query73	737	629	343	343
query74	8923	8888	8787	8787
query75	4059	3206	2726	2726
query76	3637	1194	762	762
query77	793	365	286	286
query78	10104	10262	9340	9340
query79	2082	832	564	564
query80	602	524	433	433
query81	488	255	219	219
query82	470	129	97	97
query83	178	173	164	164
query84	249	90	79	79
query85	801	424	302	302
query86	424	309	277	277
query87	4510	4637	4281	4281
query88	3623	2213	2209	2209
query89	385	311	273	273
query90	1828	206	211	206
query91	144	142	111	111
query92	85	62	64	62
query93	1619	1067	578	578
query94	673	410	305	305
query95	357	274	262	262
query96	483	562	274	274
query97	3134	3215	3133	3133
query98	228	218	205	205
query99	1427	1399	1249	1249
Total cold run time: 274643 ms
Total hot run time: 186266 ms

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

ClickBench: Total hot run time: 31.15 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 27256d3f49982211a73d122a0927bbeb43c5615c, data reload: false

query1	0.04	0.04	0.04
query2	0.12	0.10	0.11
query3	0.25	0.19	0.19
query4	1.59	0.19	0.11
query5	0.56	0.54	0.59
query6	1.22	0.72	0.71
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.58	0.52	0.52
query10	0.59	0.59	0.57
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.59
query14	2.78	2.71	2.70
query15	0.92	0.84	0.87
query16	0.38	0.37	0.38
query17	1.02	1.00	1.05
query18	0.22	0.20	0.20
query19	1.96	1.88	1.85
query20	0.01	0.01	0.01
query21	15.40	0.88	0.54
query22	0.77	1.07	0.64
query23	15.06	1.38	0.60
query24	6.96	1.34	0.86
query25	0.54	0.26	0.18
query26	0.60	0.15	0.14
query27	0.06	0.05	0.05
query28	10.21	0.88	0.42
query29	12.62	4.03	3.32
query30	0.25	0.09	0.06
query31	2.81	0.58	0.39
query32	3.22	0.54	0.49
query33	3.03	3.00	3.03
query34	15.77	5.10	4.53
query35	4.55	4.49	4.50
query36	0.66	0.50	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.2 s
Total hot run time: 31.15 s

@morningman morningman changed the title [opt](jindofs) remove unused hadoop-aliyun [fix](jindofs) add jindofs ubuntu jar in build.sh Mar 27, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 27, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Mar 27, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 27, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit 8660611 into apache:master Mar 27, 2025
29 of 30 checks passed
@morningman morningman mentioned this pull request Apr 9, 2025
Merged
yiguolei pushed a commit that referenced this pull request Apr 10, 2025
@morningman
[deps](jindofs) add jindofs for 2.1 (#49259)(#49371)(#49374)(#49502)(#…
826cbfd 
…49503)(#49886)(#49903) (#49908)

bp (#49259)(#49371)(#49374)(#49502)(#49503)(#49886)(#49903)
morningman added a commit to morningman/doris that referenced this pull request Apr 22, 2025
@morningman
[deps](jindofs) add jindofs for 2.1 (apache#49259)(apache#49371)(apac…
79bb9b1 
…he#49374)(apache#49502)(apache#49503)(apache#49886)(apache#49903) (apache#49908)

bp (apache#49259)(apache#49371)(apache#49374)(apache#49502)(apache#49503)(apache#49886)(apache#49903)
morningman added a commit to morningman/doris that referenced this pull request Apr 22, 2025
@morningman
[fix](jindofs) add jindofs ubuntu jar in build.sh (apache#49503)
c957899 
### What problem does this PR solve?

Followup apache#49502
1. add jindofs ubuntu jar in build.sh, for ubuntu 22 OS.
2. set `fs.oss.impl` after check if it is oss-hdfs, to avoid depending
on both `com.aliyun.jindodata.oss.JindoOssFileSystem` and
`org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem`
@morningman morningman mentioned this pull request Apr 22, 2025
Merged
dataroaring pushed a commit that referenced this pull request Apr 23, 2025
@morningman @hello-stephen
[deps](jindofs) add jindofs for 2.1 (#49259)(#49371)(#49374)(#49502)(#…
8aff230 
…49503)(#49886)(#49903)(#49963) (#50283)

bp (#49259)(#49371)(#49374)(#49502)(#49503)(#49886)(#49903)(#49963)

---------

Co-authored-by: Dongyang Li <lidongyang@selectdb.com>
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@morningman @koarz
[fix](jindofs) add jindofs ubuntu jar in build.sh (apache#49503)
70e299d 
### What problem does this PR solve?

Followup apache#49502
1. add jindofs ubuntu jar in build.sh, for ubuntu 22 OS.
2. set `fs.oss.impl` after check if it is oss-hdfs, to avoid depending
on both `com.aliyun.jindodata.oss.JindoOssFileSystem` and
`org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@wuwenchi wuwenchi wuwenchi approved these changes

+1 more reviewer

@suxiaogang223 suxiaogang223 suxiaogang223 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.10-merged dev/3.0.6-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@morningman @hello-stephen @doris-robot @CalvinKirs @wuwenchi @suxiaogang223 @yiguolei @dataroaring