Skip to content

branch-3.1: [fix](paimon) Fix Paimon DLF catalog caching issue by adding dlf.catalog.id to cache key #55875 #55888

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Sep 11, 2025
Merged

branch-3.1: [fix](paimon) Fix Paimon DLF catalog caching issue by adding dlf.catalog.id to cache key #55875 #55888

morrySnow merged 1 commit into from
Sep 11, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Sep 10, 2025

Cherry-picked from #55875

@zy-kkk
[fix](paimon) Fix Paimon DLF catalog caching issue by adding dlf.cata…
e61f892 
…log.id to cache key (#55875)

### What problem does this PR solve?

#### Problem:
Paimon's CachedClientPool uses a static cache with keys based on
clientClassName, metastore.uris, and metastore type. For DLF catalogs,
all these
values are identical, causing different DLF catalogs with different
dlf.catalog_id configurations to incorrectly share the same HMS client
pool.
This results in the last created catalog's configuration overriding
previous ones.

#### Root Cause:
The cache key construction in CachedClientPool.extractKey() doesn't
include DLF-specific configuration differences. Multiple catalogs with
different
dlf.catalog_id values generate identical cache keys, leading to client
pool pollution.

#### Solution:
Add dlf.catalog_id to the cache key by configuring
client-pool-cache.keys = "conf:dlf.catalog.id" in
PaimonAliyunDLFMetaStoreProperties.appendCustomCatalogOptions(). This
ensures each DLF catalog with a unique catalog_id gets its own HMS
client
  pool.
@github-actions github-actions bot requested a review from morrySnow as a code owner September 10, 2025 17:22
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Sep 10, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 10, 2025

TPC-H: Total hot run time: 32310 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e61f89220f8bcb13ea8aea1101100600b8a7db9d, data reload: false

------ Round 1 ----------------------------------
q1	17601	5467	5431	5431
q2	2026	379	279	279
q3	12029	1227	724	724
q4	10222	867	441	441
q5	8065	2332	2117	2117
q6	183	163	133	133
q7	903	740	609	609
q8	9325	1408	1108	1108
q9	5206	5009	4897	4897
q10	6778	2261	1786	1786
q11	474	277	256	256
q12	334	350	206	206
q13	17772	3583	2989	2989
q14	230	221	209	209
q15	515	449	464	449
q16	409	420	373	373
q17	600	863	358	358
q18	6900	6444	6283	6283
q19	1213	946	540	540
q20	322	332	197	197
q21	2855	2105	1935	1935
q22	1047	1004	990	990
Total cold run time: 105009 ms
Total hot run time: 32310 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5519	5472	5466	5466
q2	243	329	233	233
q3	2237	2611	2348	2348
q4	1430	1845	1444	1444
q5	4509	4977	5020	4977
q6	175	169	126	126
q7	2081	1974	1912	1912
q8	2630	2850	2670	2670
q9	7342	7237	7208	7208
q10	3012	3315	2762	2762
q11	561	487	486	486
q12	674	785	633	633
q13	3391	3798	3166	3166
q14	294	288	257	257
q15	528	460	469	460
q16	429	464	451	451
q17	1215	1728	1292	1292
q18	7685	7389	7276	7276
q19	784	1106	1071	1071
q20	1996	2068	1898	1898
q21	5308	4986	4628	4628
q22	1038	1038	975	975
Total cold run time: 53081 ms
Total hot run time: 51739 ms

@doris-robot
Copy link

doris-robot commented Sep 10, 2025

TPC-DS: Total hot run time: 191644 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e61f89220f8bcb13ea8aea1101100600b8a7db9d, data reload: false

query1	948	404	406	404
query2	6193	1931	1854	1854
query3	8675	204	207	204
query4	33481	23515	23358	23358
query5	3665	595	449	449
query6	282	184	174	174
query7	4218	486	332	332
query8	304	233	236	233
query9	9471	2664	2637	2637
query10	479	322	263	263
query11	18039	15439	15134	15134
query12	154	108	105	105
query13	1559	534	407	407
query14	10068	7007	6583	6583
query15	254	190	172	172
query16	8051	665	497	497
query17	1584	771	607	607
query18	2185	422	332	332
query19	254	189	164	164
query20	136	123	119	119
query21	205	129	114	114
query22	4707	4646	4482	4482
query23	35357	34015	34506	34015
query24	7445	2725	2745	2725
query25	568	507	450	450
query26	1228	293	195	195
query27	2135	477	371	371
query28	5808	2241	2212	2212
query29	801	618	473	473
query30	238	192	159	159
query31	1031	947	861	861
query32	99	63	60	60
query33	508	384	312	312
query34	774	867	521	521
query35	772	814	741	741
query36	1034	1058	960	960
query37	108	95	72	72
query38	4044	4032	3886	3886
query39	1532	1516	1458	1458
query40	226	115	108	108
query41	50	49	51	49
query42	133	119	108	108
query43	507	514	485	485
query44	1347	849	842	842
query45	186	175	170	170
query46	893	1058	670	670
query47	1983	1983	1956	1956
query48	400	417	346	346
query49	777	485	402	402
query50	672	688	440	440
query51	7193	7381	7237	7237
query52	109	105	94	94
query53	232	258	189	189
query54	550	532	479	479
query55	92	79	82	79
query56	274	276	259	259
query57	1278	1246	1204	1204
query58	262	222	220	220
query59	3142	3198	3103	3103
query60	308	292	275	275
query61	113	127	114	114
query62	812	738	709	709
query63	245	202	204	202
query64	4515	995	651	651
query65	3386	3301	3320	3301
query66	1367	412	309	309
query67	16232	15704	15463	15463
query68	7514	812	538	538
query69	491	308	273	273
query70	1176	1149	1071	1071
query71	373	296	268	268
query72	5728	3788	3818	3788
query73	650	741	357	357
query74	10224	9105	8984	8984
query75	3175	3147	2669	2669
query76	2962	1180	787	787
query77	555	376	274	274
query78	10349	10474	9544	9544
query79	3569	843	591	591
query80	656	519	432	432
query81	496	260	222	222
query82	586	120	93	93
query83	166	167	145	145
query84	233	103	87	87
query85	786	352	302	302
query86	385	312	310	310
query87	4386	4279	4301	4279
query88	4997	2425	2408	2408
query89	404	331	288	288
query90	1797	188	189	188
query91	146	135	111	111
query92	70	56	55	55
query93	2035	876	532	532
query94	682	411	301	301
query95	339	277	262	262
query96	498	613	284	284
query97	3196	3344	3153	3153
query98	229	210	204	204
query99	1586	1443	1285	1285
Total cold run time: 296117 ms
Total hot run time: 191644 ms

@doris-robot
Copy link

doris-robot commented Sep 10, 2025

ClickBench: Total hot run time: 29.2 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e61f89220f8bcb13ea8aea1101100600b8a7db9d, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.62	0.11	0.11
query5	0.52	0.51	0.51
query6	1.13	0.72	0.74
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.57	0.50	0.50
query10	0.54	0.54	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.60	0.60	0.58
query14	0.77	0.79	0.78
query15	0.84	0.82	0.84
query16	0.39	0.41	0.39
query17	1.07	0.99	0.95
query18	0.24	0.22	0.22
query19	1.93	1.82	1.84
query20	0.01	0.01	0.01
query21	15.39	0.95	0.59
query22	0.74	0.78	0.60
query23	15.20	1.49	0.58
query24	3.40	1.51	1.19
query25	0.25	0.11	0.13
query26	0.40	0.14	0.14
query27	0.04	0.05	0.05
query28	13.60	0.91	0.45
query29	12.56	3.96	3.26
query30	0.25	0.09	0.07
query31	2.83	0.59	0.37
query32	3.23	0.54	0.45
query33	2.98	3.05	3.04
query34	16.45	5.20	4.63
query35	4.62	4.57	4.57
query36	0.65	0.52	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.03	0.03	0.04
Total cold run time: 103.99 s
Total hot run time: 29.2 s

@morrySnow morrySnow merged commit 55a717e into branch-3.1 Sep 11, 2025
23 checks passed
@github-actions github-actions bot deleted the auto-pick-55875-branch-3.1 branch September 11, 2025 02:41
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hello-stephen @doris-robot @morrySnow @dataroaring @zy-kkk