Skip to content

[fix](load) fix S3 load cannot use HTTPS protocol for loading data #51219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
liaoxin01 merged 1 commit into from
Jun 17, 2025
Merged

[fix](load) fix S3 load cannot use HTTPS protocol for loading data #51219

liaoxin01 merged 1 commit into from
Jun 17, 2025

Conversation

@liaoxin01
Copy link
Contributor

@liaoxin01 liaoxin01 commented May 25, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #

Problem Summary:
The issue is in the checkEndpoint method where we need to preserve the original protocol (HTTP/HTTPS) when validating the endpoint.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented May 25, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liaoxin01 liaoxin01 mentioned this pull request May 28, 2025
Merged
16 tasks
dataroaring pushed a commit that referenced this pull request May 30, 2025
@liaoxin01
[fix](load) fix S3 load cannot use HTTPS protocol for loading data (#…
458b06d 
…51246)

Related PR: #51219
dataroaring
dataroaring previously approved these changes May 30, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Jun 5, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 5, 2025

TPC-H: Total hot run time: 34323 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f9d30a313b3cf2bde6143b9fec34dc5e987070e, data reload: false

------ Round 1 ----------------------------------
q1	26154	5051	4995	4995
q2	1942	274	184	184
q3	10484	1248	743	743
q4	10274	1006	527	527
q5	7542	2349	2386	2349
q6	186	170	137	137
q7	917	763	640	640
q8	9321	1288	1032	1032
q9	6911	5054	5058	5054
q10	6879	2311	1921	1921
q11	495	293	296	293
q12	356	364	221	221
q13	17796	3736	3560	3560
q14	224	221	210	210
q15	555	505	483	483
q16	434	433	377	377
q17	616	860	364	364
q18	7439	7112	7024	7024
q19	1596	952	591	591
q20	335	345	222	222
q21	3770	3193	2434	2434
q22	1057	990	962	962
Total cold run time: 115283 ms
Total hot run time: 34323 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5230	5033	5235	5033
q2	236	318	216	216
q3	2214	2683	2315	2315
q4	1373	1773	1367	1367
q5	4507	4436	4424	4424
q6	214	170	131	131
q7	2040	1943	1803	1803
q8	2604	2674	2583	2583
q9	7171	7156	7019	7019
q10	2997	3213	2786	2786
q11	595	508	480	480
q12	733	771	606	606
q13	3546	3889	3320	3320
q14	281	314	268	268
q15	519	496	480	480
q16	459	499	450	450
q17	1137	1589	1387	1387
q18	7655	7552	7332	7332
q19	850	829	922	829
q20	2059	2032	1870	1870
q21	4966	4585	4456	4456
q22	1102	1060	1018	1018
Total cold run time: 52488 ms
Total hot run time: 50173 ms

@doris-robot
Copy link

doris-robot commented Jun 5, 2025

TPC-DS: Total hot run time: 191929 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f9d30a313b3cf2bde6143b9fec34dc5e987070e, data reload: false

query1	1415	1065	1037	1037
query2	6299	1793	1777	1777
query3	10996	4482	4554	4482
query4	55324	25371	23083	23083
query5	5185	515	452	452
query6	342	203	216	203
query7	4954	502	289	289
query8	292	226	213	213
query9	5934	2632	2664	2632
query10	428	324	271	271
query11	15093	14988	14886	14886
query12	164	107	104	104
query13	1084	524	413	413
query14	10209	6336	6411	6336
query15	205	197	182	182
query16	7058	674	515	515
query17	1110	722	566	566
query18	1521	400	308	308
query19	194	181	164	164
query20	136	123	122	122
query21	207	132	115	115
query22	4258	4598	4287	4287
query23	34382	33805	33387	33387
query24	6751	2430	2457	2430
query25	466	503	428	428
query26	682	278	154	154
query27	2262	515	346	346
query28	2976	2162	2146	2146
query29	560	588	439	439
query30	275	214	198	198
query31	876	874	800	800
query32	83	64	60	60
query33	466	362	309	309
query34	802	861	550	550
query35	803	837	738	738
query36	943	1034	896	896
query37	113	101	75	75
query38	4247	4323	4271	4271
query39	1535	1462	1477	1462
query40	208	122	118	118
query41	60	56	56	56
query42	134	117	109	109
query43	502	489	487	487
query44	1347	866	851	851
query45	181	175	169	169
query46	863	1033	656	656
query47	1853	1862	1795	1795
query48	390	429	328	328
query49	678	507	415	415
query50	665	684	423	423
query51	4177	4292	4225	4225
query52	115	111	103	103
query53	239	274	189	189
query54	573	575	524	524
query55	87	90	85	85
query56	314	318	300	300
query57	1176	1196	1105	1105
query58	285	278	292	278
query59	2643	2739	2643	2643
query60	352	340	326	326
query61	155	149	152	149
query62	753	736	670	670
query63	225	195	194	194
query64	1539	1072	678	678
query65	4261	4121	4159	4121
query66	732	402	305	305
query67	16005	15557	15523	15523
query68	7405	824	529	529
query69	540	305	266	266
query70	1197	1127	1158	1127
query71	501	310	302	302
query72	5969	4786	4875	4786
query73	1399	653	352	352
query74	9221	9199	8708	8708
query75	3821	3204	2699	2699
query76	4289	1185	741	741
query77	628	372	284	284
query78	10151	10254	9269	9269
query79	2770	794	569	569
query80	698	582	425	425
query81	489	254	221	221
query82	443	123	93	93
query83	372	245	236	236
query84	308	111	89	89
query85	809	356	326	326
query86	412	305	286	286
query87	4374	4421	4246	4246
query88	3479	2282	2263	2263
query89	398	316	281	281
query90	1799	205	201	201
query91	141	143	110	110
query92	76	58	65	58
query93	2068	929	579	579
query94	658	418	305	305
query95	364	286	293	286
query96	495	566	283	283
query97	2699	2841	2604	2604
query98	239	212	206	206
query99	1416	1384	1248	1248
Total cold run time: 300770 ms
Total hot run time: 191929 ms

@doris-robot
Copy link

doris-robot commented Jun 5, 2025

ClickBench: Total hot run time: 28.51 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2f9d30a313b3cf2bde6143b9fec34dc5e987070e, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.12
query3	0.25	0.19	0.19
query4	1.59	0.19	0.20
query5	0.46	0.47	0.45
query6	1.17	0.66	0.66
query7	0.02	0.02	0.02
query8	0.06	0.03	0.04
query9	0.59	0.52	0.52
query10	0.58	0.59	0.57
query11	0.16	0.10	0.11
query12	0.15	0.12	0.11
query13	0.62	0.59	0.60
query14	0.80	0.82	0.81
query15	0.88	0.85	0.86
query16	0.37	0.38	0.38
query17	1.03	1.08	1.05
query18	0.23	0.20	0.21
query19	1.94	1.86	1.79
query20	0.02	0.01	0.01
query21	15.39	0.93	0.54
query22	0.76	1.02	0.63
query23	15.14	1.36	0.60
query24	7.53	0.82	0.32
query25	0.31	0.07	0.15
query26	0.69	0.17	0.15
query27	0.05	0.06	0.05
query28	8.57	0.88	0.44
query29	12.58	3.90	3.27
query30	0.26	0.08	0.06
query31	2.82	0.58	0.39
query32	3.23	0.56	0.47
query33	3.04	3.08	3.18
query34	15.89	5.11	4.46
query35	4.55	4.52	4.51
query36	0.68	0.50	0.48
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.07 s
Total hot run time: 28.51 s

dataroaring
dataroaring approved these changes Jun 11, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jun 11, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jun 11, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jun 11, 2025

PR approved by anyone and no changes requested.

sollhui
sollhui approved these changes Jun 17, 2025
View reviewed changes
Copy link
Contributor

@sollhui sollhui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit 02b430e into apache:master Jun 17, 2025
26 of 27 checks passed
@liaoxin01 liaoxin01 deleted the fix-https branch June 17, 2025 09:52
@liaoxin01 liaoxin01 mentioned this pull request Sep 16, 2025
Merged
16 tasks
morrySnow pushed a commit that referenced this pull request Sep 19, 2025
@liaoxin01
[fix](load) fix s3 load connection check failed (#56123)
d349c76 
### What problem does this PR solve?

PR #51219 fixed the issue of S3 load HTTPS access failure, but the
changes were accidentally dropped during the cherry-pick in PR #52479.
Now we need to add those changes back.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

+1 more reviewer

@sollhui sollhui sollhui approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.6-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@liaoxin01 @Thearas @doris-robot @sollhui @dataroaring