Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](rowcount) refresh external table's rowcount async #32997

Merged
merged 1 commit into from
Mar 31, 2024

Conversation

morningman
Copy link
Contributor

@morningman morningman commented Mar 28, 2024

Proposed changes

In previous implementation, the row count cache will be expired after 10min(by default),
and after expiration, the next row count request will miss the cache, causing unstable query plan.

In this PR, the cache will be refreshed after Config.external_cache_expire_time_minutes_after_access,
so that the cache entry will remain fresh.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37447 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f8840aede1c9298630207b545b3996a29bf53fa, data reload: false

------ Round 1 ----------------------------------
q1	17613	4203	4066	4066
q2	2100	148	151	148
q3	10592	1101	1135	1101
q4	10234	766	773	766
q5	7460	2857	2880	2857
q6	202	126	128	126
q7	1009	607	566	566
q8	9338	1986	1967	1967
q9	6885	6355	6303	6303
q10	8466	3458	3582	3458
q11	435	238	230	230
q12	389	205	205	205
q13	17800	2902	2887	2887
q14	246	209	219	209
q15	523	483	468	468
q16	513	377	388	377
q17	1066	610	593	593
q18	7118	6482	6407	6407
q19	2809	1407	1483	1407
q20	546	245	246	245
q21	3543	2867	2773	2773
q22	350	290	288	288
Total cold run time: 109237 ms
Total hot run time: 37447 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4151	4115	4084	4084
q2	329	227	227	227
q3	2929	2847	2904	2847
q4	1798	1645	1590	1590
q5	5290	5441	5402	5402
q6	200	121	122	121
q7	2298	1956	1943	1943
q8	3183	3318	3343	3318
q9	8755	8672	8608	8608
q10	3825	3939	4022	3939
q11	562	482	454	454
q12	758	610	627	610
q13	17920	3122	3100	3100
q14	311	276	259	259
q15	521	469	481	469
q16	510	460	484	460
q17	1797	1544	1522	1522
q18	8077	7621	7616	7616
q19	4863	1581	1538	1538
q20	1979	1859	1867	1859
q21	4942	4881	4924	4881
q22	520	457	471	457
Total cold run time: 75518 ms
Total hot run time: 55304 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183183 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f8840aede1c9298630207b545b3996a29bf53fa, data reload: false

query1	912	350	346	346
query2	6324	2175	2032	2032
query3	6712	218	211	211
query4	31440	21340	21248	21248
query5	4244	396	387	387
query6	266	183	179	179
query7	4639	286	293	286
query8	230	175	180	175
query9	9203	2281	2302	2281
query10	437	255	285	255
query11	15442	14216	14304	14216
query12	132	88	89	88
query13	1624	425	451	425
query14	10034	7679	7868	7679
query15	248	194	181	181
query16	8219	262	268	262
query17	1954	594	569	569
query18	2125	285	289	285
query19	350	153	157	153
query20	96	87	84	84
query21	203	136	129	129
query22	5056	4793	4809	4793
query23	33636	32834	33246	32834
query24	11044	2980	2957	2957
query25	599	407	408	407
query26	754	155	168	155
query27	2471	383	378	378
query28	6397	1899	1949	1899
query29	918	657	686	657
query30	310	152	154	152
query31	1023	782	782	782
query32	103	54	53	53
query33	780	248	262	248
query34	1285	514	521	514
query35	897	719	720	719
query36	1054	946	927	927
query37	98	67	64	64
query38	3785	3715	3614	3614
query39	1653	1590	1589	1589
query40	180	116	115	115
query41	50	45	44	44
query42	111	97	106	97
query43	526	472	472	472
query44	1200	781	776	776
query45	283	263	282	263
query46	1108	714	730	714
query47	1975	1933	1905	1905
query48	433	358	366	358
query49	874	335	335	335
query50	793	385	380	380
query51	6950	6908	6812	6812
query52	106	95	94	94
query53	356	281	282	281
query54	308	242	243	242
query55	95	85	77	77
query56	250	236	241	236
query57	1278	1187	1196	1187
query58	232	235	232	232
query59	2991	2539	2602	2539
query60	260	247	252	247
query61	114	126	109	109
query62	607	443	452	443
query63	304	280	281	280
query64	5081	4033	4092	4033
query65	3063	3022	3032	3022
query66	798	360	369	360
query67	15443	15108	15201	15108
query68	8423	552	544	544
query69	638	403	401	401
query70	1222	1144	1189	1144
query71	514	280	255	255
query72	6349	2522	2362	2362
query73	769	324	323	323
query74	7088	6469	6374	6374
query75	3624	2254	2196	2196
query76	5226	925	878	878
query77	608	254	257	254
query78	11024	10209	10058	10058
query79	10879	541	533	533
query80	2113	391	371	371
query81	521	212	217	212
query82	808	84	89	84
query83	203	146	146	146
query84	285	80	77	77
query85	1281	313	311	311
query86	416	295	291	291
query87	3742	3551	3591	3551
query88	4831	2391	2383	2383
query89	509	364	366	364
query90	1959	173	177	173
query91	171	136	136	136
query92	62	45	46	45
query93	6889	498	486	486
query94	1052	175	176	175
query95	409	299	302	299
query96	618	276	270	270
query97	2695	2483	2466	2466
query98	227	218	212	212
query99	1183	835	841	835
Total cold run time: 308804 ms
Total hot run time: 183183 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 9f8840aede1c9298630207b545b3996a29bf53fa with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.8 seconds inserted 10000000 Rows, about 724K ops/s

Jibing-Li
Jibing-Li previously approved these changes Mar 29, 2024
Copy link
Contributor

@Jibing-Li Jibing-Li left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38668 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5a3a9965d4506132081961050d1e4a6a43c69f1d, data reload: false

------ Round 1 ----------------------------------
q1	17775	4562	4330	4330
q2	2716	202	202	202
q3	11286	1233	1160	1160
q4	10297	769	779	769
q5	7829	2818	2741	2741
q6	222	137	139	137
q7	1068	622	633	622
q8	9326	2079	2062	2062
q9	7712	6615	6532	6532
q10	8556	3538	3583	3538
q11	480	243	244	243
q12	410	226	216	216
q13	18898	2976	2979	2976
q14	266	230	243	230
q15	511	474	477	474
q16	531	406	394	394
q17	963	569	613	569
q18	7311	6659	6684	6659
q19	1576	1403	1454	1403
q20	707	320	321	320
q21	3639	2834	2780	2780
q22	371	311	322	311
Total cold run time: 112450 ms
Total hot run time: 38668 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4269	4154	4174	4154
q2	390	285	273	273
q3	2953	2720	2687	2687
q4	1861	1519	1532	1519
q5	5299	5196	5244	5196
q6	210	123	126	123
q7	2259	1880	1874	1874
q8	3188	3306	3277	3277
q9	8695	8580	8628	8580
q10	3890	3690	3691	3690
q11	570	498	488	488
q12	763	609	603	603
q13	14049	3009	2981	2981
q14	314	260	280	260
q15	503	473	476	473
q16	478	426	422	422
q17	1776	1431	1419	1419
q18	7577	7374	7311	7311
q19	1607	1491	1530	1491
q20	1965	1757	1741	1741
q21	5207	4830	4854	4830
q22	545	471	472	471
Total cold run time: 68368 ms
Total hot run time: 53863 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 180923 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5a3a9965d4506132081961050d1e4a6a43c69f1d, data reload: false

query1	882	366	1121	366
query2	6483	1989	1898	1898
query3	6664	216	216	216
query4	37564	21532	21281	21281
query5	4258	398	390	390
query6	228	176	180	176
query7	3986	309	292	292
query8	227	182	185	182
query9	5440	2303	2290	2290
query10	483	238	258	238
query11	14574	14166	14298	14166
query12	144	92	92	92
query13	985	377	375	375
query14	9516	6875	6816	6816
query15	220	179	182	179
query16	7256	273	276	273
query17	1689	601	572	572
query18	1606	293	288	288
query19	208	157	160	157
query20	93	91	89	89
query21	202	134	134	134
query22	4982	4811	4807	4807
query23	33336	32884	32611	32611
query24	11996	2827	2759	2759
query25	608	394	384	384
query26	1825	155	156	155
query27	3310	323	329	323
query28	7371	1860	1836	1836
query29	915	635	614	614
query30	291	158	154	154
query31	895	731	731	731
query32	67	58	59	58
query33	558	277	252	252
query34	888	482	490	482
query35	908	674	710	674
query36	1040	904	914	904
query37	286	72	70	70
query38	3525	3388	3397	3388
query39	1599	1563	1534	1534
query40	287	132	129	129
query41	51	47	47	47
query42	108	97	95	95
query43	492	452	479	452
query44	1192	739	724	724
query45	281	267	266	266
query46	1105	688	680	680
query47	1925	1869	1856	1856
query48	390	311	304	304
query49	1176	366	365	365
query50	772	395	395	395
query51	6867	6791	6627	6627
query52	100	93	88	88
query53	346	283	286	283
query54	268	248	243	243
query55	85	86	80	80
query56	256	235	230	230
query57	1224	1141	1118	1118
query58	239	210	209	209
query59	2784	2547	2658	2547
query60	276	244	250	244
query61	128	111	110	110
query62	650	436	449	436
query63	313	276	286	276
query64	6090	4115	4105	4105
query65	3117	3034	3044	3034
query66	1354	351	356	351
query67	15473	15000	14824	14824
query68	10485	536	541	536
query69	617	330	316	316
query70	1240	1135	1160	1135
query71	528	285	271	271
query72	6954	2620	2450	2450
query73	1083	326	337	326
query74	6795	6336	6392	6336
query75	3869	2304	2340	2304
query76	6534	1052	1049	1049
query77	624	252	251	251
query78	10981	10339	10211	10211
query79	12200	536	529	529
query80	2277	424	411	411
query81	503	1080	212	212
query82	749	87	90	87
query83	223	174	172	172
query84	260	83	82	82
query85	1199	278	263	263
query86	395	283	295	283
query87	3740	3548	3523	3523
query88	5470	2419	2405	2405
query89	514	372	381	372
query90	2012	184	185	184
query91	139	98	101	98
query92	57	48	48	48
query93	6341	511	508	508
query94	1278	182	180	180
query95	410	309	306	306
query96	603	271	273	271
query97	2670	2526	2439	2439
query98	247	222	221	221
query99	1124	820	822	820
Total cold run time: 316578 ms
Total hot run time: 180923 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.12 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5a3a9965d4506132081961050d1e4a6a43c69f1d, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.24	0.05	0.04
query4	1.68	0.07	0.07
query5	0.49	0.48	0.49
query6	1.13	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.55	0.50	0.49
query10	0.54	0.55	0.55
query11	0.16	0.11	0.12
query12	0.14	0.12	0.11
query13	0.59	0.60	0.59
query14	0.77	0.79	0.77
query15	0.82	0.82	0.83
query16	0.37	0.35	0.35
query17	0.94	1.03	0.97
query18	0.23	0.25	0.22
query19	1.79	1.68	1.66
query20	0.01	0.02	0.01
query21	15.54	0.65	0.64
query22	2.43	2.26	2.53
query23	17.18	0.99	0.81
query24	1.16	0.25	0.18
query25	0.09	0.08	0.08
query26	0.22	0.17	0.17
query27	0.08	0.07	0.08
query28	14.11	0.94	0.92
query29	12.51	3.26	3.28
query30	0.29	0.10	0.09
query31	2.80	0.37	0.40
query32	3.27	0.46	0.45
query33	2.85	2.87	2.89
query34	16.47	4.49	4.43
query35	4.46	4.50	4.43
query36	0.62	0.46	0.46
query37	0.17	0.15	0.16
query38	0.14	0.15	0.14
query39	0.05	0.04	0.04
query40	0.18	0.15	0.15
query41	0.09	0.05	0.05
query42	0.06	0.06	0.05
query43	0.04	0.04	0.04
Total cold run time: 105.44 s
Total hot run time: 30.12 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 5a3a9965d4506132081961050d1e4a6a43c69f1d with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.3 seconds inserted 10000000 Rows, about 699K ops/s

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38258 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 17fa9f44f2420ef51b15b631a3e929c7ec5e93c4, data reload: false

------ Round 1 ----------------------------------
q1	17614	4341	4272	4272
q2	2073	199	189	189
q3	10440	1181	1159	1159
q4	10200	740	716	716
q5	7473	2758	2756	2756
q6	213	131	131	131
q7	1028	594	601	594
q8	9221	2000	2052	2000
q9	7427	6613	6541	6541
q10	8461	3478	3541	3478
q11	451	244	231	231
q12	522	214	217	214
q13	17787	2963	2976	2963
q14	268	235	234	234
q15	523	477	475	475
q16	530	389	378	378
q17	959	653	565	565
q18	7376	6725	6644	6644
q19	7546	1481	1431	1431
q20	672	318	304	304
q21	3455	2686	2943	2686
q22	356	297	300	297
Total cold run time: 114595 ms
Total hot run time: 38258 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4351	4229	4197	4197
q2	381	272	282	272
q3	3003	2677	2687	2677
q4	1794	1530	1535	1530
q5	5275	5146	5222	5146
q6	212	122	129	122
q7	2199	1847	1879	1847
q8	3216	3282	3305	3282
q9	8546	8532	8530	8530
q10	4083	3862	3854	3854
q11	636	511	487	487
q12	753	638	614	614
q13	16390	3303	3211	3211
q14	307	307	283	283
q15	519	466	481	466
q16	501	442	443	442
q17	1804	1546	1481	1481
q18	8109	8088	7785	7785
q19	1705	1523	1535	1523
q20	1966	1839	1857	1839
q21	5302	4932	4900	4900
q22	521	472	475	472
Total cold run time: 71573 ms
Total hot run time: 54960 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183530 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 17fa9f44f2420ef51b15b631a3e929c7ec5e93c4, data reload: false

query1	1207	1125	1115	1115
query2	7039	2094	1902	1902
query3	6645	220	222	220
query4	24474	21389	21466	21389
query5	4236	425	409	409
query6	275	191	188	188
query7	4610	303	302	302
query8	224	180	178	178
query9	8492	2307	2278	2278
query10	417	242	255	242
query11	15477	14232	14250	14232
query12	148	91	99	91
query13	1635	368	375	368
query14	9377	7059	6845	6845
query15	221	176	185	176
query16	7071	278	273	273
query17	1457	599	556	556
query18	1791	285	293	285
query19	207	160	163	160
query20	94	90	89	89
query21	200	143	137	137
query22	4971	4799	4817	4799
query23	33409	32699	32770	32699
query24	7857	2933	2867	2867
query25	590	417	415	415
query26	718	167	167	167
query27	2905	359	368	359
query28	4261	1942	1883	1883
query29	878	626	620	620
query30	296	154	164	154
query31	988	738	786	738
query32	92	57	58	57
query33	436	257	249	249
query34	902	483	497	483
query35	862	729	727	727
query36	1040	933	921	921
query37	115	70	68	68
query38	3823	3649	3641	3641
query39	1614	1593	1596	1593
query40	189	141	128	128
query41	51	48	45	45
query42	116	106	109	106
query43	510	453	457	453
query44	1091	768	760	760
query45	297	267	268	267
query46	1105	717	713	713
query47	2030	1919	1961	1919
query48	402	328	319	319
query49	776	382	371	371
query50	816	414	410	410
query51	6865	6740	6760	6740
query52	101	99	100	99
query53	377	297	294	294
query54	295	267	266	266
query55	83	80	88	80
query56	259	239	254	239
query57	1266	1200	1200	1200
query58	248	228	223	223
query59	3000	2858	2557	2557
query60	268	243	251	243
query61	114	113	113	113
query62	542	439	433	433
query63	312	292	283	283
query64	4830	4162	4116	4116
query65	3051	3066	3047	3047
query66	868	338	329	329
query67	15659	15130	15119	15119
query68	9282	536	542	536
query69	589	308	315	308
query70	1422	1152	1184	1152
query71	466	268	287	268
query72	6950	2635	2453	2453
query73	1634	320	325	320
query74	6945	6347	6356	6347
query75	4156	2345	2384	2345
query76	5118	1070	1037	1037
query77	580	256	245	245
query78	11089	10292	10253	10253
query79	12628	539	534	534
query80	2266	441	426	426
query81	510	224	213	213
query82	320	95	90	90
query83	231	168	172	168
query84	268	83	83	83
query85	902	282	272	272
query86	343	293	295	293
query87	3704	3583	3542	3542
query88	5576	2477	2399	2399
query89	510	374	390	374
query90	2221	182	179	179
query91	126	99	108	99
query92	62	48	49	48
query93	6736	506	504	504
query94	1473	187	183	183
query95	407	317	306	306
query96	603	268	273	268
query97	2643	2490	2525	2490
query98	228	217	221	217
query99	1055	831	832	831
Total cold run time: 297618 ms
Total hot run time: 183530 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.43 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 17fa9f44f2420ef51b15b631a3e929c7ec5e93c4, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.67	0.07	0.08
query5	0.49	0.49	0.50
query6	1.15	0.65	0.65
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.51
query10	0.56	0.53	0.55
query11	0.16	0.10	0.11
query12	0.14	0.11	0.10
query13	0.59	0.57	0.58
query14	0.78	0.80	0.78
query15	0.84	0.80	0.80
query16	0.37	0.38	0.38
query17	0.93	1.01	1.01
query18	0.22	0.21	0.22
query19	1.89	1.81	1.64
query20	0.02	0.01	0.01
query21	15.53	0.66	0.65
query22	2.24	2.62	1.68
query23	17.15	0.98	0.83
query24	1.18	0.22	0.20
query25	0.08	0.08	0.09
query26	0.22	0.17	0.17
query27	0.08	0.08	0.07
query28	14.09	0.94	0.92
query29	12.52	3.19	3.20
query30	0.28	0.08	0.09
query31	2.82	0.38	0.38
query32	3.27	0.45	0.46
query33	2.84	2.84	2.79
query34	16.54	4.40	4.43
query35	4.47	4.49	4.49
query36	0.60	0.47	0.46
query37	0.17	0.14	0.14
query38	0.14	0.14	0.15
query39	0.05	0.05	0.04
query40	0.18	0.14	0.16
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.05	0.03	0.04
Total cold run time: 105.42 s
Total hot run time: 29.43 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 17fa9f44f2420ef51b15b631a3e929c7ec5e93c4 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.9 seconds inserted 10000000 Rows, about 719K ops/s

Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 30, 2024
@morningman morningman merged commit 10f6eb0 into apache:master Mar 31, 2024
28 of 30 checks passed
morningman added a commit to morningman/doris that referenced this pull request Mar 31, 2024
In previous implementation, the row count cache will be expired after 10min(by default),
and after expiration, the next row count request will miss the cache, causing unstable query plan.

In this PR, the cache will be refreshed after Config.external_cache_expire_time_minutes_after_access,
so that the cache entry will remain fresh.
morningman added a commit to morningman/doris that referenced this pull request Apr 7, 2024
In previous implementation, the row count cache will be expired after 10min(by default),
and after expiration, the next row count request will miss the cache, causing unstable query plan.

In this PR, the cache will be refreshed after Config.external_cache_expire_time_minutes_after_access,
so that the cache entry will remain fresh.
@yiguolei yiguolei mentioned this pull request Apr 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.8-merged dev/2.1.2-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants