Skip to content

[fix](create-resource) fix potential concurrent modification exception when creating resource #50356

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 2 commits into from
Apr 26, 2025
Merged

[fix](create-resource) fix potential concurrent modification exception when creating resource #50356

morningman merged 2 commits into from
Apr 26, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Apr 23, 2025 •
edited
Loading

What problem does this PR solve?

Related PR: #48424

Problem Summary:

The CreateResourceCommandTest may throw concurrent modification exception.
This is because in JdbcResource, the origin property map may be modified.

This PR changes；

  1. Change the origin property map to ImmutableMap, to avoid potential exception.
  2. Copy the map to the jdbc resource to avoid modifying the origin map

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Apr 23, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

morningman commented Apr 23, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 23, 2025

TPC-H: Total hot run time: 33870 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 42ae946ce836dd2f2d019b948492a7e70f74f556, data reload: false

------ Round 1 ----------------------------------
q1	26134	4993	4999	4993
q2	2082	283	189	189
q3	10457	1242	700	700
q4	10243	1022	538	538
q5	7630	2337	2418	2337
q6	182	165	133	133
q7	904	758	608	608
q8	9327	1289	1031	1031
q9	6799	5110	5097	5097
q10	6866	2290	1885	1885
q11	483	284	260	260
q12	355	363	227	227
q13	17753	3711	3141	3141
q14	222	236	224	224
q15	530	480	481	480
q16	450	452	401	401
q17	603	857	369	369
q18	7516	7188	7041	7041
q19	1857	945	570	570
q20	328	324	216	216
q21	3982	3535	2425	2425
q22	1049	1009	1005	1005
Total cold run time: 115752 ms
Total hot run time: 33870 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5205	5112	5114	5112
q2	235	337	233	233
q3	2135	2662	2268	2268
q4	1452	1859	1432	1432
q5	4432	4353	4457	4353
q6	216	172	126	126
q7	2011	1925	1736	1736
q8	2604	2526	2521	2521
q9	7254	7282	7017	7017
q10	3023	3153	2744	2744
q11	572	499	500	499
q12	690	780	622	622
q13	3561	3900	3251	3251
q14	274	313	287	287
q15	535	481	470	470
q16	457	489	456	456
q17	1169	1512	1420	1420
q18	7687	7566	7490	7490
q19	789	872	1016	872
q20	2007	1973	1827	1827
q21	5231	4934	4953	4934
q22	1100	1065	988	988
Total cold run time: 52639 ms
Total hot run time: 50658 ms

@doris-robot
Copy link

doris-robot commented Apr 23, 2025

TPC-DS: Total hot run time: 191958 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 42ae946ce836dd2f2d019b948492a7e70f74f556, data reload: false

query1	1414	1083	1059	1059
query2	6238	1798	1808	1798
query3	11103	4615	4641	4615
query4	53820	25270	23096	23096
query5	5291	530	474	474
query6	393	222	206	206
query7	5219	509	285	285
query8	339	255	239	239
query9	6526	2566	2597	2566
query10	428	320	252	252
query11	15247	15029	14825	14825
query12	149	104	102	102
query13	1170	496	377	377
query14	10029	6176	6221	6176
query15	198	189	183	183
query16	6896	651	537	537
query17	1070	744	571	571
query18	1516	393	343	343
query19	202	191	178	178
query20	146	126	117	117
query21	213	134	109	109
query22	4453	4491	4244	4244
query23	34180	33472	33182	33182
query24	6570	2382	2394	2382
query25	455	475	401	401
query26	617	272	159	159
query27	2229	505	335	335
query28	3205	2178	2172	2172
query29	557	547	430	430
query30	281	220	175	175
query31	873	854	760	760
query32	72	69	64	64
query33	478	371	314	314
query34	754	930	522	522
query35	850	823	765	765
query36	961	1003	892	892
query37	113	103	71	71
query38	4157	4245	4219	4219
query39	1500	1423	1410	1410
query40	211	120	109	109
query41	63	57	52	52
query42	122	113	109	109
query43	506	515	482	482
query44	1323	819	821	819
query45	185	173	169	169
query46	849	1107	642	642
query47	1862	1860	1833	1833
query48	374	425	300	300
query49	704	497	409	409
query50	658	678	422	422
query51	4285	4168	4234	4168
query52	106	113	101	101
query53	231	270	187	187
query54	602	582	523	523
query55	82	82	92	82
query56	306	316	282	282
query57	1170	1225	1170	1170
query58	285	288	263	263
query59	2666	2802	2625	2625
query60	345	326	319	319
query61	159	152	151	151
query62	733	747	680	680
query63	237	194	185	185
query64	1521	1125	821	821
query65	4489	4332	4237	4237
query66	753	404	309	309
query67	15827	15499	15411	15411
query68	7646	892	516	516
query69	527	313	263	263
query70	1211	1130	1061	1061
query71	502	320	288	288
query72	5812	4637	4589	4589
query73	1100	582	348	348
query74	8874	9191	8992	8992
query75	3759	3185	2705	2705
query76	4316	1188	744	744
query77	624	368	278	278
query78	9926	10106	9247	9247
query79	1805	802	570	570
query80	623	562	449	449
query81	485	263	220	220
query82	433	124	96	96
query83	375	256	238	238
query84	290	100	78	78
query85	829	365	317	317
query86	377	310	288	288
query87	4343	4423	4366	4366
query88	2890	2196	2237	2196
query89	409	311	281	281
query90	1953	210	214	210
query91	139	137	111	111
query92	74	56	56	56
query93	1166	945	581	581
query94	693	419	288	288
query95	365	297	272	272
query96	491	544	275	275
query97	3209	3250	3091	3091
query98	230	212	194	194
query99	1445	1423	1285	1285
Total cold run time: 297487 ms
Total hot run time: 191958 ms

@doris-robot
Copy link

doris-robot commented Apr 23, 2025

ClickBench: Total hot run time: 29.01 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 42ae946ce836dd2f2d019b948492a7e70f74f556, data reload: false

query1	0.04	0.03	0.02
query2	0.12	0.10	0.10
query3	0.25	0.19	0.20
query4	1.62	0.20	0.20
query5	0.61	0.59	0.58
query6	1.18	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.57	0.52	0.51
query10	0.56	0.55	0.56
query11	0.16	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.60
query14	1.19	1.16	1.19
query15	0.88	0.86	0.84
query16	0.39	0.39	0.38
query17	1.01	1.05	1.03
query18	0.21	0.19	0.19
query19	1.87	1.81	1.81
query20	0.01	0.02	0.02
query21	15.40	0.94	0.56
query22	0.74	1.19	0.63
query23	14.98	1.36	0.62
query24	7.34	2.15	0.39
query25	0.39	0.35	0.07
query26	0.55	0.17	0.14
query27	0.06	0.04	0.05
query28	9.05	0.90	0.43
query29	12.56	4.00	3.26
query30	0.25	0.10	0.06
query31	2.82	0.59	0.38
query32	3.24	0.54	0.46
query33	2.93	2.97	3.07
query34	15.82	5.17	4.56
query35	4.57	4.55	4.48
query36	0.68	0.50	0.48
query37	0.09	0.07	0.07
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.37 s
Total hot run time: 29.01 s

starocean999
starocean999 previously approved these changes Apr 24, 2025
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 24, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Apr 24, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 24, 2025

PR approved by anyone and no changes requested.

@morningman morningman changed the title [fix](create-resource) fix potential concurrent modification exception when craeting resource [fix](create-resource) fix potential concurrent modification exception when creating resource Apr 25, 2025
@morningman
Copy link
Contributor Author

morningman commented Apr 25, 2025

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 25, 2025
@doris-robot
Copy link

doris-robot commented Apr 25, 2025

TPC-H: Total hot run time: 34090 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 30338255b729f2d50121432b2ce1cdbd29114a60, data reload: false

------ Round 1 ----------------------------------
q1	25639	5330	4998	4998
q2	2063	275	177	177
q3	10400	1228	674	674
q4	10230	1003	524	524
q5	7531	2351	2341	2341
q6	174	165	137	137
q7	935	753	629	629
q8	9335	1276	1121	1121
q9	6822	5104	5071	5071
q10	6805	2318	1898	1898
q11	472	283	276	276
q12	352	346	212	212
q13	17781	3638	3018	3018
q14	243	232	210	210
q15	525	482	494	482
q16	444	462	406	406
q17	607	858	378	378
q18	7614	7436	7320	7320
q19	1721	946	590	590
q20	332	340	223	223
q21	4280	3443	2439	2439
q22	1064	1042	966	966
Total cold run time: 115369 ms
Total hot run time: 34090 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5550	5075	5085	5075
q2	246	323	233	233
q3	2148	2623	2273	2273
q4	1379	1774	1432	1432
q5	4412	4407	4408	4407
q6	214	172	129	129
q7	2026	1936	1789	1789
q8	2561	2699	2510	2510
q9	7322	7186	7220	7186
q10	2942	3158	2770	2770
q11	569	515	499	499
q12	729	793	632	632
q13	3481	3910	3285	3285
q14	273	305	270	270
q15	534	488	477	477
q16	499	520	464	464
q17	1145	1533	1402	1402
q18	7620	7680	7527	7527
q19	804	815	890	815
q20	1916	1952	1835	1835
q21	5379	4859	4773	4773
q22	1100	1078	1040	1040
Total cold run time: 52849 ms
Total hot run time: 50823 ms

@doris-robot
Copy link

doris-robot commented Apr 25, 2025

TPC-DS: Total hot run time: 192187 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 30338255b729f2d50121432b2ce1cdbd29114a60, data reload: false

query1	1399	1090	1046	1046
query2	6055	1841	1903	1841
query3	11129	4555	4586	4555
query4	52204	24203	22992	22992
query5	5223	575	444	444
query6	320	190	213	190
query7	4884	498	288	288
query8	305	246	241	241
query9	5578	2581	2593	2581
query10	448	324	269	269
query11	15051	15007	14853	14853
query12	161	111	109	109
query13	1054	516	403	403
query14	10038	6328	6313	6313
query15	211	201	179	179
query16	7101	700	515	515
query17	1142	737	582	582
query18	1518	397	315	315
query19	199	179	166	166
query20	126	124	122	122
query21	212	129	107	107
query22	4394	4523	4306	4306
query23	34234	33639	33350	33350
query24	6718	2430	2438	2430
query25	483	458	411	411
query26	676	280	156	156
query27	2354	503	347	347
query28	2970	2161	2141	2141
query29	581	555	432	432
query30	275	223	189	189
query31	855	863	769	769
query32	73	63	62	62
query33	471	384	313	313
query34	767	860	528	528
query35	788	908	755	755
query36	954	972	909	909
query37	113	105	78	78
query38	4365	4272	4299	4272
query39	1518	1506	1428	1428
query40	217	122	111	111
query41	58	55	52	52
query42	122	111	109	109
query43	519	517	497	497
query44	1315	834	806	806
query45	182	174	166	166
query46	850	1048	652	652
query47	1854	1893	1823	1823
query48	390	436	301	301
query49	709	504	434	434
query50	644	694	419	419
query51	4147	4245	4125	4125
query52	117	109	98	98
query53	221	268	197	197
query54	573	575	525	525
query55	82	87	82	82
query56	320	298	279	279
query57	1209	1186	1140	1140
query58	263	262	259	259
query59	2770	2932	2750	2750
query60	334	347	308	308
query61	160	145	150	145
query62	742	742	725	725
query63	231	187	186	186
query64	1595	1136	783	783
query65	4374	4248	4176	4176
query66	810	397	305	305
query67	15749	15520	15210	15210
query68	6927	872	516	516
query69	535	308	265	265
query70	1166	1083	1114	1083
query71	495	315	282	282
query72	5866	4743	4843	4743
query73	1484	639	342	342
query74	9300	8915	8933	8915
query75	4061	3242	2719	2719
query76	4241	1206	745	745
query77	737	370	281	281
query78	9951	10115	9234	9234
query79	2093	815	563	563
query80	608	497	429	429
query81	484	248	218	218
query82	422	133	100	100
query83	332	260	227	227
query84	288	107	84	84
query85	788	355	320	320
query86	368	297	280	280
query87	4393	4480	4270	4270
query88	3190	2237	2229	2229
query89	465	317	286	286
query90	1975	210	217	210
query91	139	146	107	107
query92	76	60	54	54
query93	1150	917	587	587
query94	665	417	315	315
query95	366	295	277	277
query96	489	565	273	273
query97	3167	3270	3134	3134
query98	223	206	205	205
query99	1427	1402	1244	1244
Total cold run time: 295375 ms
Total hot run time: 192187 ms

@doris-robot
Copy link

doris-robot commented Apr 25, 2025

ClickBench: Total hot run time: 29.7 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 30338255b729f2d50121432b2ce1cdbd29114a60, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.10	0.11
query3	0.25	0.19	0.19
query4	1.58	0.20	0.20
query5	0.62	0.59	0.61
query6	1.18	0.71	0.71
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.54	0.51
query10	0.56	0.55	0.58
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.60
query14	1.16	1.19	1.20
query15	0.87	0.84	0.85
query16	0.38	0.39	0.40
query17	1.07	1.04	1.04
query18	0.21	0.21	0.19
query19	1.91	1.81	1.80
query20	0.01	0.01	0.02
query21	15.39	0.87	0.54
query22	0.75	1.13	0.64
query23	15.03	1.41	0.58
query24	6.70	2.00	1.01
query25	0.48	0.10	0.06
query26	0.58	0.16	0.13
query27	0.05	0.05	0.06
query28	9.86	0.82	0.43
query29	12.53	3.95	3.27
query30	0.25	0.09	0.06
query31	2.81	0.58	0.38
query32	3.24	0.55	0.46
query33	3.16	3.03	3.06
query34	15.66	5.12	4.49
query35	4.57	4.56	4.61
query36	0.68	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.03
query40	0.18	0.14	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.77 s
Total hot run time: 29.7 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 26, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Apr 26, 2025

PR approved by at least one committer and no changes requested.

@morningman morningman merged commit 8d1ddea into apache:master Apr 26, 2025
30 of 32 checks passed
morningman added a commit to morningman/doris that referenced this pull request Apr 29, 2025
@morningman
[fix](create-resource) fix potential concurrent modification exceptio…
38b58b6 
…n when creating resource (apache#50356)

Related PR: apache#48424

Problem Summary:

The `CreateResourceCommandTest` may throw concurrent modification
exception.
This is because in `JdbcResource`, the origin property map may be
modified.

This PR changes；
1. Change the origin property map to ImmutableMap, to avoid potential
exception.
2. Copy the map to the jdbc resource to avoid modifying the origin map
@morningman morningman mentioned this pull request Apr 29, 2025
Merged
@yagagagaga yagagagaga mentioned this pull request May 6, 2025
Merged
16 tasks
gavinchou pushed a commit that referenced this pull request May 7, 2025
@yagagagaga
[fix](vault) fix UnsupportedOperationException if no use_path_style…
210b960 
… in CreateStorageVaultStmt (#50615)

Related PR: #50356 #45155

Problem Summary:

```sql
CREATE STORAGE VAULT test PROPERTIES
    (
      "type"="S3",
      "s3.endpoint"="cos.ap-guangzhou.myqcloud.com",
      "s3.access_key" = "*****",
      "s3.secret_key" = "*****",
      "s3.region" = "ap-guangzhou",
      "s3.root.path" = "prefix",
      "s3.bucket" = "test",
      "provider" = "COS"
    )
```

throw Exception:
```java
2025-05-06 15:05:16,395 WARN (mysql-nio-pool-0|235) [StmtExecutor.handleDdlStmt():3216] DDL statement(CREATE STORAGE VAULT test PROPERTIES
    (
      "type"="S3",
      "s3.endpoint"="cos.ap-guangzhou.myqcloud.com",
      "s3.access_key" = "*****",
      "s3.secret_key" = "*****",
      "s3.region" = "ap-guangzhou",
      "s3.root.path" = "prefix",
      "s3.bucket" = "test",
      "provider" = "COS"
    )) process failed.
java.lang.UnsupportedOperationException: null
        at com.google.common.collect.ImmutableMap.put(ImmutableMap.java:814) ~[guava-33.2.1-jre.jar:?]
        at org.apache.doris.catalog.StorageVault.getStorageVaultInstance(StorageVault.java:151) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.StorageVault.fromStmt(StorageVault.java:103) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.StorageVaultMgr.createStorageVaultResource(StorageVaultMgr.java:69) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.DdlExecutor.execute(DdlExecutor.java:440) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.handleDdlStmt(StmtExecutor.java:3195) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:1124) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:644) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:574) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:559) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:349) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:249) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:233) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:261) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:444) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
```
yagagagaga added a commit to yagagagaga/doris that referenced this pull request May 7, 2025
@yagagagaga
[fix](vault) fix UnsupportedOperationException if no use_path_style…
4a0a436 
… in CreateStorageVaultStmt (apache#50615)

Related PR: apache#50356 apache#45155

Problem Summary:

```sql
CREATE STORAGE VAULT test PROPERTIES
    (
      "type"="S3",
      "s3.endpoint"="cos.ap-guangzhou.myqcloud.com",
      "s3.access_key" = "*****",
      "s3.secret_key" = "*****",
      "s3.region" = "ap-guangzhou",
      "s3.root.path" = "prefix",
      "s3.bucket" = "test",
      "provider" = "COS"
    )
```

throw Exception:
```java
2025-05-06 15:05:16,395 WARN (mysql-nio-pool-0|235) [StmtExecutor.handleDdlStmt():3216] DDL statement(CREATE STORAGE VAULT test PROPERTIES
    (
      "type"="S3",
      "s3.endpoint"="cos.ap-guangzhou.myqcloud.com",
      "s3.access_key" = "*****",
      "s3.secret_key" = "*****",
      "s3.region" = "ap-guangzhou",
      "s3.root.path" = "prefix",
      "s3.bucket" = "test",
      "provider" = "COS"
    )) process failed.
java.lang.UnsupportedOperationException: null
        at com.google.common.collect.ImmutableMap.put(ImmutableMap.java:814) ~[guava-33.2.1-jre.jar:?]
        at org.apache.doris.catalog.StorageVault.getStorageVaultInstance(StorageVault.java:151) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.StorageVault.fromStmt(StorageVault.java:103) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.StorageVaultMgr.createStorageVaultResource(StorageVaultMgr.java:69) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.DdlExecutor.execute(DdlExecutor.java:440) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.handleDdlStmt(StmtExecutor.java:3195) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:1124) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:644) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:574) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:559) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:349) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:249) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:233) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:261) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:444) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
```

(cherry picked from commit 210b960)
morningman added a commit to morningman/doris that referenced this pull request May 7, 2025
@morningman
[fix](create-resource) fix potential concurrent modification exceptio…
5a0167d 
…n when creating resource (apache#50356)

Related PR: apache#48424

Problem Summary:

The `CreateResourceCommandTest` may throw concurrent modification
exception.
This is because in `JdbcResource`, the origin property map may be
modified.

This PR changes；
1. Change the origin property map to ImmutableMap, to avoid potential
exception.
2. Copy the map to the jdbc resource to avoid modifying the origin map
morningman pushed a commit to morningman/doris that referenced this pull request May 7, 2025
@yagagagaga @morningman
[fix](vault) fix UnsupportedOperationException if no use_path_style…
4d25e92 
… in CreateStorageVaultStmt (apache#50615)

Related PR: apache#50356 apache#45155

Problem Summary:

```sql
CREATE STORAGE VAULT test PROPERTIES
    (
      "type"="S3",
      "s3.endpoint"="cos.ap-guangzhou.myqcloud.com",
      "s3.access_key" = "*****",
      "s3.secret_key" = "*****",
      "s3.region" = "ap-guangzhou",
      "s3.root.path" = "prefix",
      "s3.bucket" = "test",
      "provider" = "COS"
    )
```

throw Exception:
```java
2025-05-06 15:05:16,395 WARN (mysql-nio-pool-0|235) [StmtExecutor.handleDdlStmt():3216] DDL statement(CREATE STORAGE VAULT test PROPERTIES
    (
      "type"="S3",
      "s3.endpoint"="cos.ap-guangzhou.myqcloud.com",
      "s3.access_key" = "*****",
      "s3.secret_key" = "*****",
      "s3.region" = "ap-guangzhou",
      "s3.root.path" = "prefix",
      "s3.bucket" = "test",
      "provider" = "COS"
    )) process failed.
java.lang.UnsupportedOperationException: null
        at com.google.common.collect.ImmutableMap.put(ImmutableMap.java:814) ~[guava-33.2.1-jre.jar:?]
        at org.apache.doris.catalog.StorageVault.getStorageVaultInstance(StorageVault.java:151) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.StorageVault.fromStmt(StorageVault.java:103) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.StorageVaultMgr.createStorageVaultResource(StorageVaultMgr.java:69) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.DdlExecutor.execute(DdlExecutor.java:440) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.handleDdlStmt(StmtExecutor.java:3195) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:1124) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:644) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:574) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:559) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:349) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:249) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:233) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:261) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:444) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
```
morningman added a commit to morningman/doris that referenced this pull request May 7, 2025
@morningman
[fix](create-resource) fix potential concurrent modification exceptio…
76fd857 
…n when creating resource (apache#50356)

Related PR: apache#48424

Problem Summary:

The `CreateResourceCommandTest` may throw concurrent modification
exception.
This is because in `JdbcResource`, the origin property map may be
modified.

This PR changes；
1. Change the origin property map to ImmutableMap, to avoid potential
exception.
2. Copy the map to the jdbc resource to avoid modifying the origin map
morningman pushed a commit to morningman/doris that referenced this pull request May 7, 2025
@yagagagaga @morningman
[fix](vault) fix UnsupportedOperationException if no use_path_style…
6c52b12 
… in CreateStorageVaultStmt (apache#50615)

Related PR: apache#50356 apache#45155

Problem Summary:

```sql
CREATE STORAGE VAULT test PROPERTIES
    (
      "type"="S3",
      "s3.endpoint"="cos.ap-guangzhou.myqcloud.com",
      "s3.access_key" = "*****",
      "s3.secret_key" = "*****",
      "s3.region" = "ap-guangzhou",
      "s3.root.path" = "prefix",
      "s3.bucket" = "test",
      "provider" = "COS"
    )
```

throw Exception:
```java
2025-05-06 15:05:16,395 WARN (mysql-nio-pool-0|235) [StmtExecutor.handleDdlStmt():3216] DDL statement(CREATE STORAGE VAULT test PROPERTIES
    (
      "type"="S3",
      "s3.endpoint"="cos.ap-guangzhou.myqcloud.com",
      "s3.access_key" = "*****",
      "s3.secret_key" = "*****",
      "s3.region" = "ap-guangzhou",
      "s3.root.path" = "prefix",
      "s3.bucket" = "test",
      "provider" = "COS"
    )) process failed.
java.lang.UnsupportedOperationException: null
        at com.google.common.collect.ImmutableMap.put(ImmutableMap.java:814) ~[guava-33.2.1-jre.jar:?]
        at org.apache.doris.catalog.StorageVault.getStorageVaultInstance(StorageVault.java:151) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.StorageVault.fromStmt(StorageVault.java:103) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.StorageVaultMgr.createStorageVaultResource(StorageVaultMgr.java:69) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.DdlExecutor.execute(DdlExecutor.java:440) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.handleDdlStmt(StmtExecutor.java:3195) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:1124) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:644) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:574) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:559) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:349) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:249) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:233) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:261) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:444) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
```
morningman added a commit to morningman/doris that referenced this pull request May 14, 2025
@morningman
[fix](create-resource) fix potential concurrent modification exceptio…
1c5ff56 
…n when creating resource (apache#50356)

Related PR: apache#48424

Problem Summary:

The `CreateResourceCommandTest` may throw concurrent modification
exception.
This is because in `JdbcResource`, the origin property map may be
modified.

This PR changes；
1. Change the origin property map to ImmutableMap, to avoid potential
exception.
2. Copy the map to the jdbc resource to avoid modifying the origin map
morningman pushed a commit to morningman/doris that referenced this pull request May 14, 2025
@yagagagaga @morningman
[fix](vault) fix UnsupportedOperationException if no use_path_style…
d3608b4 
… in CreateStorageVaultStmt (apache#50615)

Related PR: apache#50356 apache#45155

Problem Summary:

```sql
CREATE STORAGE VAULT test PROPERTIES
    (
      "type"="S3",
      "s3.endpoint"="cos.ap-guangzhou.myqcloud.com",
      "s3.access_key" = "*****",
      "s3.secret_key" = "*****",
      "s3.region" = "ap-guangzhou",
      "s3.root.path" = "prefix",
      "s3.bucket" = "test",
      "provider" = "COS"
    )
```

throw Exception:
```java
2025-05-06 15:05:16,395 WARN (mysql-nio-pool-0|235) [StmtExecutor.handleDdlStmt():3216] DDL statement(CREATE STORAGE VAULT test PROPERTIES
    (
      "type"="S3",
      "s3.endpoint"="cos.ap-guangzhou.myqcloud.com",
      "s3.access_key" = "*****",
      "s3.secret_key" = "*****",
      "s3.region" = "ap-guangzhou",
      "s3.root.path" = "prefix",
      "s3.bucket" = "test",
      "provider" = "COS"
    )) process failed.
java.lang.UnsupportedOperationException: null
        at com.google.common.collect.ImmutableMap.put(ImmutableMap.java:814) ~[guava-33.2.1-jre.jar:?]
        at org.apache.doris.catalog.StorageVault.getStorageVaultInstance(StorageVault.java:151) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.StorageVault.fromStmt(StorageVault.java:103) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.StorageVaultMgr.createStorageVaultResource(StorageVaultMgr.java:69) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.DdlExecutor.execute(DdlExecutor.java:440) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.handleDdlStmt(StmtExecutor.java:3195) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:1124) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:644) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:574) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:559) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:349) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:249) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:233) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:261) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:444) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
```
dataroaring pushed a commit that referenced this pull request May 15, 2025
@morningman @yagagagaga
branch-3.0: [fix](create-resource) fix potential concurrent modificat…
3ae4211 
…ion exception when creating resource (#50356, #50615) (#50533)

bp #50356 #50615

---------

Co-authored-by: yagagagaga <zhangminke@selectdb.com>
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@morningman @koarz
[fix](create-resource) fix potential concurrent modification exceptio…
39dfdef 
…n when creating resource (apache#50356)

### What problem does this PR solve?

Related PR: apache#48424

Problem Summary:

The `CreateResourceCommandTest` may throw concurrent modification
exception.
This is because in `JdbcResource`, the origin property map may be
modified.

This PR changes；
1. Change the origin property map to ImmutableMap, to avoid potential
exception.
2. Copy the map to the jdbc resource to avoid modifying the origin map
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@yagagagaga @koarz
[fix](vault) fix UnsupportedOperationException if no use_path_style…
dc3c412 
… in CreateStorageVaultStmt (apache#50615)

Related PR: apache#50356 apache#45155

Problem Summary:

```sql
CREATE STORAGE VAULT test PROPERTIES
    (
      "type"="S3",
      "s3.endpoint"="cos.ap-guangzhou.myqcloud.com",
      "s3.access_key" = "*****",
      "s3.secret_key" = "*****",
      "s3.region" = "ap-guangzhou",
      "s3.root.path" = "prefix",
      "s3.bucket" = "test",
      "provider" = "COS"
    )
```

throw Exception:
```java
2025-05-06 15:05:16,395 WARN (mysql-nio-pool-0|235) [StmtExecutor.handleDdlStmt():3216] DDL statement(CREATE STORAGE VAULT test PROPERTIES
    (
      "type"="S3",
      "s3.endpoint"="cos.ap-guangzhou.myqcloud.com",
      "s3.access_key" = "*****",
      "s3.secret_key" = "*****",
      "s3.region" = "ap-guangzhou",
      "s3.root.path" = "prefix",
      "s3.bucket" = "test",
      "provider" = "COS"
    )) process failed.
java.lang.UnsupportedOperationException: null
        at com.google.common.collect.ImmutableMap.put(ImmutableMap.java:814) ~[guava-33.2.1-jre.jar:?]
        at org.apache.doris.catalog.StorageVault.getStorageVaultInstance(StorageVault.java:151) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.StorageVault.fromStmt(StorageVault.java:103) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.StorageVaultMgr.createStorageVaultResource(StorageVaultMgr.java:69) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.DdlExecutor.execute(DdlExecutor.java:440) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.handleDdlStmt(StmtExecutor.java:3195) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:1124) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:644) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:574) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:559) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:349) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:249) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:233) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:261) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:444) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@wuwenchi wuwenchi wuwenchi approved these changes

@starocean999 starocean999 starocean999 approved these changes

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

@w41ter w41ter Awaiting requested review from w41ter w41ter is a code owner

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.6-merged p0_test reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@morningman @Thearas @doris-robot @gavinchou @CalvinKirs @wuwenchi @starocean999 @dataroaring