Skip to content

[opt](deps) add jindofs in classpath after hadoop libs #51689

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Jun 14, 2025
Merged

[opt](deps) add jindofs in classpath after hadoop libs #51689

morningman merged 1 commit into from
Jun 14, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Jun 12, 2025 •
edited
Loading

What problem does this PR solve?

Related PR: #49903

Problem Summary:
We should add jindofs jars in CLASSPATH, but after hadoop jars.
So that we can use the right hadoop deps instead of deps from jindofs.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jun 12, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

morningman commented Jun 12, 2025

run buildall

4 similar comments
@morningman
Copy link
Contributor Author

morningman commented Jun 12, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 13, 2025

run buildall

@morningman
Copy link
Contributor Author

morningman commented Jun 13, 2025

run buildall

@morningman
Copy link
Contributor Author

morningman commented Jun 13, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 13, 2025

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 13, 2025

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.40% (15072/26724)
Line Coverage 45.17% (134764/298369)
Region Coverage 44.28% (67821/153162)
Branch Coverage 38.84% (34801/89592)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 13, 2025

BE Regression && UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 60.89% (16014/26301)
Line Coverage 50.47% (150507/298211)
Region Coverage 47.72% (85969/180135)
Branch Coverage 41.28% (42271/102398)

@morningman
Copy link
Contributor Author

morningman commented Jun 13, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 13, 2025

TPC-H: Total hot run time: 34535 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 98712bac2acc00a540de52c6102947552710f164, data reload: false

------ Round 1 ----------------------------------
q1	17606	5175	4932	4932
q2	1940	286	184	184
q3	10390	1259	744	744
q4	10233	1039	549	549
q5	7700	2354	2383	2354
q6	183	165	132	132
q7	935	765	619	619
q8	9347	1316	1067	1067
q9	6822	5116	5163	5116
q10	6865	2330	1897	1897
q11	483	285	280	280
q12	355	359	215	215
q13	17775	3696	3088	3088
q14	231	229	215	215
q15	564	484	480	480
q16	426	438	375	375
q17	599	870	366	366
q18	7873	7274	7092	7092
q19	1438	943	595	595
q20	342	332	228	228
q21	3742	3219	3023	3023
q22	1050	1029	984	984
Total cold run time: 106899 ms
Total hot run time: 34535 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5046	4998	5008	4998
q2	241	329	217	217
q3	2153	2652	2313	2313
q4	1372	1835	1354	1354
q5	4245	4146	4417	4146
q6	210	170	131	131
q7	2005	1925	1764	1764
q8	2605	2543	2467	2467
q9	7282	7219	7231	7219
q10	3073	3137	2738	2738
q11	598	504	491	491
q12	698	774	602	602
q13	3553	3927	3285	3285
q14	283	288	294	288
q15	534	493	493	493
q16	435	486	432	432
q17	1147	1545	1393	1393
q18	7802	7621	7620	7620
q19	818	825	921	825
q20	2065	2045	1959	1959
q21	4990	4371	4449	4371
q22	1106	1062	1050	1050
Total cold run time: 52261 ms
Total hot run time: 50156 ms

@doris-robot
Copy link

doris-robot commented Jun 13, 2025

TPC-DS: Total hot run time: 192445 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 98712bac2acc00a540de52c6102947552710f164, data reload: false

query1	1397	1037	984	984
query2	6328	1772	1758	1758
query3	11103	4659	4351	4351
query4	54914	24484	23589	23589
query5	5174	520	487	487
query6	384	213	198	198
query7	4977	526	292	292
query8	289	227	217	217
query9	6173	2661	2639	2639
query10	447	332	273	273
query11	15049	15076	14798	14798
query12	154	104	105	104
query13	1098	527	387	387
query14	10029	6255	6547	6255
query15	199	200	189	189
query16	7048	661	471	471
query17	1073	730	590	590
query18	1548	439	323	323
query19	212	208	214	208
query20	131	127	123	123
query21	204	124	109	109
query22	4389	4322	4227	4227
query23	34482	33514	33581	33514
query24	6598	2365	2398	2365
query25	460	472	405	405
query26	675	282	171	171
query27	2200	522	345	345
query28	3242	2162	2166	2162
query29	584	582	447	447
query30	269	220	191	191
query31	853	860	782	782
query32	72	66	63	63
query33	441	354	309	309
query34	771	868	524	524
query35	776	817	784	784
query36	937	1040	887	887
query37	117	101	79	79
query38	4163	4327	4231	4231
query39	1510	1483	1474	1474
query40	209	119	115	115
query41	62	56	55	55
query42	127	110	112	110
query43	523	494	476	476
query44	1377	854	859	854
query45	190	173	181	173
query46	857	1022	671	671
query47	1845	1885	1774	1774
query48	430	434	332	332
query49	648	471	408	408
query50	663	688	405	405
query51	4179	4225	4163	4163
query52	116	113	102	102
query53	237	257	182	182
query54	587	586	512	512
query55	90	90	90	90
query56	313	311	273	273
query57	1215	1234	1149	1149
query58	273	250	250	250
query59	2472	2650	2525	2525
query60	312	309	316	309
query61	122	137	150	137
query62	729	733	656	656
query63	234	187	218	187
query64	1444	1004	668	668
query65	4215	4207	4138	4138
query66	720	415	311	311
query67	15801	15488	15423	15423
query68	4469	905	562	562
query69	488	318	271	271
query70	1141	1111	1082	1082
query71	424	325	312	312
query72	5947	4984	5049	4984
query73	782	685	367	367
query74	9179	9122	8914	8914
query75	3280	3191	2693	2693
query76	3158	1282	753	753
query77	581	479	296	296
query78	10103	10137	9330	9330
query79	2674	843	583	583
query80	832	526	482	482
query81	522	253	225	225
query82	407	123	95	95
query83	255	260	239	239
query84	294	105	84	84
query85	763	344	329	329
query86	419	287	278	278
query87	4412	4398	4272	4272
query88	3852	2293	2297	2293
query89	390	315	284	284
query90	1829	214	215	214
query91	135	147	111	111
query92	77	61	58	58
query93	2666	969	595	595
query94	689	419	307	307
query95	376	300	289	289
query96	512	582	283	283
query97	2651	2731	2627	2627
query98	237	212	211	211
query99	1316	1415	1269	1269
Total cold run time: 295320 ms
Total hot run time: 192445 ms

@doris-robot
Copy link

doris-robot commented Jun 13, 2025

ClickBench: Total hot run time: 28.9 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 98712bac2acc00a540de52c6102947552710f164, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.04
query3	0.25	0.07	0.06
query4	1.62	0.10	0.10
query5	0.45	0.41	0.42
query6	1.16	0.65	0.68
query7	0.03	0.02	0.01
query8	0.04	0.04	0.04
query9	0.57	0.51	0.52
query10	0.58	0.57	0.57
query11	0.15	0.11	0.11
query12	0.15	0.12	0.11
query13	0.62	0.59	0.60
query14	0.79	0.81	0.81
query15	0.90	0.84	0.89
query16	0.40	0.38	0.39
query17	1.00	1.07	1.05
query18	0.22	0.21	0.21
query19	1.93	1.79	1.85
query20	0.02	0.01	0.01
query21	15.44	0.90	0.55
query22	0.76	1.31	0.68
query23	14.83	1.40	0.63
query24	6.64	1.84	0.75
query25	0.49	0.22	0.09
query26	0.58	0.17	0.15
query27	0.05	0.05	0.05
query28	9.47	0.85	0.46
query29	12.60	4.00	3.37
query30	0.25	0.09	0.06
query31	2.82	0.59	0.39
query32	3.23	0.54	0.46
query33	3.10	3.06	3.08
query34	15.89	5.13	4.55
query35	4.56	4.59	4.54
query36	0.68	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 102.88 s
Total hot run time: 28.9 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 13, 2025

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 13, 2025

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.40% (15074/26725)
Line Coverage 45.17% (134779/298395)
Region Coverage 44.28% (67822/153172)
Branch Coverage 38.84% (34803/89598)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 13, 2025

BE Regression && UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.77% (20982/26302)
Line Coverage 72.69% (216797/298237)
Region Coverage 70.92% (127762/180145)
Branch Coverage 64.60% (66150/102404)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 13, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jun 13, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 13, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit ef1e7cc into apache:master Jun 14, 2025
24 of 26 checks passed
morningman added a commit to morningman/doris that referenced this pull request Jun 14, 2025
@morningman
[opt](deps) add jindofs in classpath after hadoop libs (apache#51689)
62a76f6 
Related PR: apache#49903

Problem Summary:
We should add jindofs jars in CLASSPATH, but after hadoop jars.
So that we can use the right hadoop deps instead of deps from jindofs.
@morningman morningman mentioned this pull request Jun 14, 2025
Merged
dataroaring pushed a commit that referenced this pull request Jun 17, 2025
@morningman
[opt](deps) add jindofs in classpath after hadoop libs (#51689) (#51722)
c9755f5 
bp #51689
morningman added a commit to morningman/doris that referenced this pull request Jul 1, 2025
@morningman
[opt](deps) add jindofs in classpath after hadoop libs (apache#51689) (…
9f7990b 
…apache#51722)

bp apache#51689
@morningman morningman mentioned this pull request Jul 1, 2025
Merged
morningman added a commit that referenced this pull request Jul 1, 2025
@morningman
[opt](deps) add jindofs in classpath after hadoop libs (#51689) (#51722…
d89d2b9 
…) (#52589)

bp #51689
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

+1 more reviewer

@suxiaogang223 suxiaogang223 suxiaogang223 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.6-merged dev/3.1.0-merged p0_r reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@morningman @Thearas @hello-stephen @doris-robot @CalvinKirs @suxiaogang223 @dataroaring @morrySnow