Skip to content

[fix](iceberg) Add missing iceberg-aws dependency to metadata scanner #55741

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Sep 6, 2025
Merged

[fix](iceberg) Add missing iceberg-aws dependency to metadata scanner #55741

morningman merged 1 commit into from
Sep 6, 2025

Conversation

@zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Sep 6, 2025

Issue

Fixed NoSuchMethodException when accessing S3-based Iceberg tables:

  • Cannot create org.apache.iceberg.aws.ApacheHttpClientConfigurations
  • Missing ApacheHttpClientConfigurations.create method

Root Cause

The iceberg-metadata-scanner module was missing the iceberg-aws dependency, causing runtime failures when accessing S3 storage.

Solution

Added iceberg-aws dependency to iceberg-metadata-scanner/pom.xml

Why This Issue Is Not Always Reproducible

This is a classpath race condition that only occurs under specific circumstances:

  1. Environment Dependency: Only affects deployments where preload-extensions module is not loaded or iceberg-aws classes are not in runtime classpath
  2. ClassLoader Isolation: Error depends on module loading order:
    • ✅ preload-extensions loads first → classes available → no error
    • ❌ iceberg-metadata-scanner loads first → classes missing → error occurs
  3. Storage Backend Dependency: Only triggers when accessing S3-based Iceberg tables (not HDFS/local files)
  4. Runtime Loading Order: Depends on which module's ClassLoader gets ApacheHttpClientConfigurations first

@Thearas
Copy link
Contributor

Thearas commented Sep 6, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zy-kkk
Copy link
Member Author

zy-kkk commented Sep 6, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 6, 2025

TPC-H: Total hot run time: 34401 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 83e6e6ea8c8efeebe6e4fac2fde392596f7e4883, data reload: false

------ Round 1 ----------------------------------
q1	17632	5221	5040	5040
q2	2005	321	206	206
q3	10266	1259	703	703
q4	10225	1018	521	521
q5	7535	2388	2302	2302
q6	188	165	139	139
q7	935	773	630	630
q8	9348	1328	1076	1076
q9	6893	5099	5165	5099
q10	6976	2387	1951	1951
q11	493	300	279	279
q12	352	366	222	222
q13	17767	3613	3032	3032
q14	236	245	220	220
q15	579	502	490	490
q16	1010	997	954	954
q17	601	861	363	363
q18	7342	7046	7073	7046
q19	1359	952	559	559
q20	338	348	235	235
q21	3673	2540	2349	2349
q22	1060	1042	985	985
Total cold run time: 106813 ms
Total hot run time: 34401 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5179	5088	5095	5088
q2	255	332	233	233
q3	2144	2659	2255	2255
q4	1334	1748	1331	1331
q5	4216	4354	4535	4354
q6	221	170	132	132
q7	2004	1968	1790	1790
q8	2726	2761	2672	2672
q9	7275	7448	7315	7315
q10	3195	3284	2921	2921
q11	558	534	502	502
q12	690	784	877	784
q13	3387	3837	3266	3266
q14	301	309	278	278
q15	560	495	479	479
q16	1123	1125	1068	1068
q17	1200	1621	1433	1433
q18	7836	7642	7925	7642
q19	791	797	880	797
q20	2046	1964	1850	1850
q21	4676	4260	4305	4260
q22	1096	1064	999	999
Total cold run time: 52813 ms
Total hot run time: 51449 ms

@doris-robot
Copy link

doris-robot commented Sep 6, 2025

TPC-DS: Total hot run time: 189003 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 83e6e6ea8c8efeebe6e4fac2fde392596f7e4883, data reload: false

query1	1098	465	407	407
query2	6563	1690	1710	1690
query3	6749	220	223	220
query4	26314	23461	22773	22773
query5	4429	658	507	507
query6	331	273	211	211
query7	4657	514	298	298
query8	326	253	250	250
query9	8629	2945	2865	2865
query10	487	367	296	296
query11	15807	15340	14997	14997
query12	183	132	118	118
query13	1670	554	420	420
query14	10218	9087	9223	9087
query15	210	197	173	173
query16	7168	672	489	489
query17	1270	796	650	650
query18	2015	448	345	345
query19	209	212	186	186
query20	136	132	125	125
query21	230	140	118	118
query22	3961	4191	4040	4040
query23	33858	33156	33045	33045
query24	8106	2373	2433	2373
query25	593	526	449	449
query26	1248	278	167	167
query27	2730	518	351	351
query28	4368	2271	2217	2217
query29	844	604	502	502
query30	288	222	200	200
query31	879	836	716	716
query32	89	90	83	83
query33	581	407	353	353
query34	804	849	528	528
query35	824	817	749	749
query36	974	1034	915	915
query37	130	116	95	95
query38	3513	3610	3448	3448
query39	1477	1433	1417	1417
query40	224	132	128	128
query41	65	66	68	66
query42	130	119	117	117
query43	515	502	477	477
query44	1345	875	866	866
query45	187	182	175	175
query46	866	1026	672	672
query47	1823	1847	1785	1785
query48	398	429	338	338
query49	754	539	416	416
query50	657	691	399	399
query51	3899	3908	3904	3904
query52	120	121	112	112
query53	254	273	196	196
query54	618	618	557	557
query55	100	91	99	91
query56	360	338	317	317
query57	1184	1192	1152	1152
query58	291	284	277	277
query59	2637	2604	2602	2602
query60	370	359	343	343
query61	169	161	163	161
query62	805	713	673	673
query63	237	196	201	196
query64	4570	1160	951	951
query65	4025	3967	3980	3967
query66	1199	458	371	371
query67	15476	15149	15131	15131
query68	8002	934	586	586
query69	519	350	304	304
query70	1349	1283	1334	1283
query71	595	351	328	328
query72	6092	4987	5033	4987
query73	654	612	358	358
query74	9264	8835	9195	8835
query75	3586	3267	2782	2782
query76	3256	1161	736	736
query77	765	406	343	343
query78	9500	9739	8897	8897
query79	2711	860	607	607
query80	744	590	543	543
query81	503	265	232	232
query82	482	169	138	138
query83	270	275	259	259
query84	259	116	96	96
query85	914	483	453	453
query86	395	319	312	312
query87	3683	3791	3662	3662
query88	3819	2276	2270	2270
query89	403	333	304	304
query90	1887	225	226	225
query91	163	162	132	132
query92	91	74	70	70
query93	2142	1014	655	655
query94	745	421	328	328
query95	467	339	335	335
query96	494	594	283	283
query97	2943	3000	2868	2868
query98	252	216	222	216
query99	1384	1416	1263	1263
Total cold run time: 275577 ms
Total hot run time: 189003 ms

@doris-robot
Copy link

doris-robot commented Sep 6, 2025

ClickBench: Total hot run time: 30.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 83e6e6ea8c8efeebe6e4fac2fde392596f7e4883, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.06	0.06
query3	0.25	0.08	0.08
query4	1.61	0.12	0.12
query5	0.29	0.26	0.25
query6	1.20	0.66	0.66
query7	0.04	0.03	0.02
query8	0.06	0.05	0.04
query9	0.60	0.52	0.53
query10	0.57	0.58	0.57
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.63	0.64	0.61
query14	1.04	1.04	1.06
query15	0.87	0.84	0.87
query16	0.40	0.41	0.40
query17	1.03	1.08	1.09
query18	0.22	0.20	0.20
query19	1.99	1.86	1.84
query20	0.01	0.01	0.01
query21	15.41	1.03	0.59
query22	0.77	1.02	0.67
query23	15.10	1.39	0.63
query24	6.93	0.94	1.53
query25	0.54	0.30	0.08
query26	0.64	0.16	0.14
query27	0.06	0.05	0.06
query28	9.88	0.89	0.44
query29	12.62	3.91	3.34
query30	0.28	0.12	0.11
query31	2.83	0.59	0.39
query32	3.23	0.56	0.49
query33	3.02	3.10	3.24
query34	16.09	5.47	4.83
query35	4.95	4.94	4.93
query36	0.73	0.51	0.50
query37	0.11	0.08	0.07
query38	0.06	0.05	0.04
query39	0.04	0.03	0.02
query40	0.20	0.17	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 104.94 s
Total hot run time: 30.22 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 6, 2025

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@github-actions
Copy link
Contributor

github-actions bot commented Sep 6, 2025

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 6, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 6, 2025

PR approved by at least one committer and no changes requested.

@morningman morningman merged commit ab70b9f into apache:master Sep 6, 2025
28 of 29 checks passed
github-actions bot pushed a commit that referenced this pull request Sep 6, 2025
@zy-kkk
[fix](iceberg) Add missing iceberg-aws dependency to metadata scanner (…
18a660a 
…#55741)

## Issue
  Fixed `NoSuchMethodException` when accessing S3-based Iceberg tables:
- `Cannot create org.apache.iceberg.aws.ApacheHttpClientConfigurations`
  - Missing `ApacheHttpClientConfigurations.create` method
@github-actions github-actions bot mentioned this pull request Sep 6, 2025
Merged
wenzhenghu pushed a commit to wenzhenghu/doris that referenced this pull request Sep 8, 2025
@zy-kkk @wenzhenghu
[fix](iceberg) Add missing iceberg-aws dependency to metadata scanner (…
e5352cc 
…apache#55741)

## Issue
  Fixed `NoSuchMethodException` when accessing S3-based Iceberg tables:
- `Cannot create org.apache.iceberg.aws.ApacheHttpClientConfigurations`
  - Missing `ApacheHttpClientConfigurations.create` method
@zy-kkk zy-kkk deleted the fix_iceberg_sys_scanner_class branch September 8, 2025 02:42
morrySnow pushed a commit that referenced this pull request Sep 8, 2025
@github-actions @zy-kkk
branch-3.1: [fix](iceberg) Add missing iceberg-aws dependency to meta…
9251979 
…data scanner #55741 (#55743)

Cherry-picked from #55741

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@CalvinKirs CalvinKirs Awaiting requested review from CalvinKirs CalvinKirs is a code owner

+1 more reviewer

@suxiaogang223 suxiaogang223 suxiaogang223 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@zy-kkk @Thearas @doris-robot @hello-stephen @morningman @suxiaogang223 @morrySnow