Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[nereids](enhance) Added order by and limit expr for show routine load #40938

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

Toms1999
Copy link
Contributor

Background

In order to improve the usability of the SHOW ROUTINE LOAD command, we have added support for ORDER BY and LIMIT expressions. This enhancement will allow users to filter and sort routine load tasks more effectively.

Changes

  • Implemented ORDER BY functionality to sort routine load tasks based on specified columns (e.g., ID, Name, etc.).
  • Added LIMIT expression to restrict the number of routine load tasks returned by the query.
  • Updated corresponding SQL parsing logic to handle these expressions.

Impact

This enhancement improves the flexibility of SHOW ROUTINE LOAD queries, making it easier to manage routine loads in large environments. Users can now quickly identify specific tasks using sorting and limiting mechanisms.

Testing

  • Unit tests added for the new ORDER BY and LIMIT functionality.
  • Verified that the changes do not affect existing behavior when ORDER BY and LIMIT are not used.
  • Manual testing performed on a local environment to ensure expected functionality.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Toms1999
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41538 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9a07b296dd74b523b6a7b3cf5fea2dc140932490, data reload: false

------ Round 1 ----------------------------------
q1	18207	7446	7371	7371
q2	2903	159	166	159
q3	11059	1147	1239	1147
q4	10944	754	673	673
q5	7843	3098	3144	3098
q6	236	151	150	150
q7	1009	639	598	598
q8	9762	1996	2036	1996
q9	6811	6383	6408	6383
q10	6994	2287	2250	2250
q11	439	247	256	247
q12	410	221	222	221
q13	17784	3003	2962	2962
q14	242	230	217	217
q15	582	515	513	513
q16	658	615	614	614
q17	954	846	826	826
q18	7226	6775	6679	6679
q19	1477	1048	968	968
q20	585	293	280	280
q21	4271	3172	3297	3172
q22	1109	1014	1041	1014
Total cold run time: 111505 ms
Total hot run time: 41538 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7250	7193	7244	7193
q2	333	225	223	223
q3	2882	2784	2755	2755
q4	1896	1682	1693	1682
q5	5382	5515	5466	5466
q6	229	143	137	137
q7	2098	1704	1746	1704
q8	3149	3298	3301	3298
q9	8328	8387	8400	8387
q10	3401	3347	3359	3347
q11	572	470	464	464
q12	769	567	593	567
q13	5500	3020	2967	2967
q14	304	278	263	263
q15	572	511	512	511
q16	688	656	659	656
q17	1761	1560	1541	1541
q18	7869	7356	7442	7356
q19	1692	1478	1360	1360
q20	2034	1811	1811	1811
q21	5412	5051	5075	5051
q22	1138	1033	1011	1011
Total cold run time: 63259 ms
Total hot run time: 57750 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194873 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9a07b296dd74b523b6a7b3cf5fea2dc140932490, data reload: false

query1	955	369	379	369
query2	6537	2102	2088	2088
query3	6707	214	216	214
query4	34431	23564	23441	23441
query5	4513	469	469	469
query6	286	178	169	169
query7	4637	305	309	305
query8	287	231	225	225
query9	9718	2675	2636	2636
query10	498	270	274	270
query11	18160	15121	15222	15121
query12	151	101	98	98
query13	1632	424	401	401
query14	10594	7320	7314	7314
query15	247	170	174	170
query16	8066	439	471	439
query17	1651	547	535	535
query18	2134	298	298	298
query19	353	147	146	146
query20	116	111	113	111
query21	213	106	101	101
query22	4501	4316	4096	4096
query23	34553	34223	33979	33979
query24	11192	2917	2869	2869
query25	663	413	407	407
query26	1199	166	160	160
query27	2349	290	287	287
query28	7688	2472	2456	2456
query29	860	430	417	417
query30	324	165	158	158
query31	1045	767	821	767
query32	100	63	61	61
query33	784	292	308	292
query34	962	495	491	491
query35	870	731	753	731
query36	1097	938	957	938
query37	149	88	92	88
query38	4120	3968	3948	3948
query39	1458	1408	1420	1408
query40	212	98	96	96
query41	53	48	50	48
query42	112	102	98	98
query43	538	484	493	484
query44	1334	831	797	797
query45	198	166	166	166
query46	1141	743	741	741
query47	1929	1806	1813	1806
query48	446	356	374	356
query49	1161	423	415	415
query50	819	408	415	408
query51	7220	6919	6902	6902
query52	101	87	89	87
query53	256	186	179	179
query54	1314	467	482	467
query55	79	77	82	77
query56	299	274	268	268
query57	1233	1078	1080	1078
query58	247	235	243	235
query59	3161	3122	2918	2918
query60	303	275	289	275
query61	126	122	127	122
query62	849	660	648	648
query63	219	189	186	186
query64	5387	727	738	727
query65	3290	3210	3170	3170
query66	1431	306	308	306
query67	16278	15763	15394	15394
query68	3115	577	581	577
query69	431	299	289	289
query70	1196	1153	1116	1116
query71	335	264	261	261
query72	6047	4025	4022	4022
query73	760	326	335	326
query74	9489	9107	9164	9107
query75	3374	2685	2645	2645
query76	2127	899	878	878
query77	429	293	295	293
query78	9874	9348	9578	9348
query79	1121	892	877	877
query80	887	599	594	594
query81	528	256	258	256
query82	904	234	230	230
query83	216	169	160	160
query84	247	111	108	108
query85	744	393	361	361
query86	379	301	321	301
query87	4449	4438	4451	4438
query88	4578	4077	4077	4077
query89	377	363	361	361
query90	1799	315	308	308
query91	163	185	166	166
query92	78	73	70	70
query93	930	912	903	903
query94	814	346	349	346
query95	455	404	404	404
query96	488	489	488	488
query97	3171	3130	3167	3130
query98	243	225	218	218
query99	1416	1347	1288	1288
Total cold run time: 296700 ms
Total hot run time: 194873 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.92 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9a07b296dd74b523b6a7b3cf5fea2dc140932490, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.52	0.49	0.52
query6	1.14	0.74	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.56	0.49	0.50
query10	0.54	0.55	0.54
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.58	0.59
query14	2.96	3.03	2.96
query15	0.90	0.83	0.82
query16	0.39	0.37	0.38
query17	0.98	1.00	1.04
query18	0.23	0.20	0.21
query19	1.94	1.84	2.03
query20	0.01	0.01	0.01
query21	15.36	0.59	0.58
query22	2.84	2.32	1.98
query23	17.66	0.76	0.85
query24	2.82	0.94	0.94
query25	0.19	0.06	0.05
query26	0.48	0.14	0.14
query27	0.05	0.05	0.04
query28	10.90	1.10	1.08
query29	12.55	3.34	3.30
query30	0.25	0.06	0.06
query31	2.89	0.39	0.39
query32	3.27	0.45	0.46
query33	2.99	2.99	3.08
query34	17.13	4.39	4.39
query35	4.44	4.48	4.50
query36	0.66	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.03
query40	0.16	0.12	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 108.1 s
Total hot run time: 32.92 s

@Toms1999
Copy link
Contributor Author

run p0

@Toms1999
Copy link
Contributor Author

run cloud_p0

@Toms1999
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 42054 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 956178e05db2df0c4213a72f27b1789abfe2100e, data reload: false

------ Round 1 ----------------------------------
q1	17953	7500	7731	7500
q2	2062	162	164	162
q3	10838	1108	1183	1108
q4	10380	842	703	703
q5	7766	3116	3099	3099
q6	234	153	147	147
q7	1010	639	623	623
q8	9448	2096	2067	2067
q9	6853	6419	6492	6419
q10	7018	2324	2329	2324
q11	436	255	250	250
q12	410	214	223	214
q13	17777	3040	3029	3029
q14	241	227	226	226
q15	582	550	511	511
q16	680	606	620	606
q17	995	837	823	823
q18	7327	6783	6834	6783
q19	1407	1059	986	986
q20	589	294	290	290
q21	4008	3185	3317	3185
q22	1124	999	1004	999
Total cold run time: 109138 ms
Total hot run time: 42054 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7308	7276	7205	7205
q2	329	238	242	238
q3	3042	2961	2938	2938
q4	2111	1852	1762	1762
q5	5645	5649	5730	5649
q6	238	149	144	144
q7	2185	1767	1784	1767
q8	3326	3434	3403	3403
q9	8865	8940	8766	8766
q10	3467	3470	3507	3470
q11	591	496	482	482
q12	825	620	604	604
q13	9332	3226	3180	3180
q14	297	274	269	269
q15	604	537	526	526
q16	709	679	712	679
q17	1833	1615	1623	1615
q18	8285	7798	7694	7694
q19	1744	1530	1410	1410
q20	2149	1878	1886	1878
q21	5554	5170	5502	5170
q22	1143	1078	1053	1053
Total cold run time: 69582 ms
Total hot run time: 59902 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195840 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 956178e05db2df0c4213a72f27b1789abfe2100e, data reload: false

query1	1234	867	888	867
query2	6405	2104	2043	2043
query3	10789	4067	3912	3912
query4	64943	26885	23531	23531
query5	5259	488	486	486
query6	414	166	196	166
query7	5531	300	298	298
query8	353	220	227	220
query9	8855	2653	2645	2645
query10	495	301	275	275
query11	17947	15248	15751	15248
query12	172	103	101	101
query13	1496	430	428	428
query14	10829	7602	7588	7588
query15	212	176	187	176
query16	6883	460	469	460
query17	1137	629	587	587
query18	1730	308	301	301
query19	209	148	164	148
query20	130	124	114	114
query21	210	102	102	102
query22	4857	4471	4773	4471
query23	34614	34210	33846	33846
query24	5949	2881	2857	2857
query25	507	409	427	409
query26	648	158	160	158
query27	1616	286	285	285
query28	4118	2447	2428	2428
query29	674	428	422	422
query30	235	151	157	151
query31	970	769	771	769
query32	73	55	59	55
query33	433	296	296	296
query34	881	486	500	486
query35	864	739	720	720
query36	1071	933	956	933
query37	144	87	91	87
query38	4001	3904	4057	3904
query39	1484	1436	1403	1403
query40	203	103	96	96
query41	50	49	48	48
query42	116	97	99	97
query43	525	486	494	486
query44	1152	814	799	799
query45	198	168	163	163
query46	1122	793	762	762
query47	1936	1792	1847	1792
query48	469	359	356	356
query49	677	388	383	383
query50	856	405	419	405
query51	7040	7043	6966	6966
query52	100	88	83	83
query53	253	176	177	176
query54	569	454	455	454
query55	76	79	75	75
query56	287	254	271	254
query57	1217	1091	1076	1076
query58	222	233	237	233
query59	3192	2929	2970	2929
query60	293	254	281	254
query61	106	104	104	104
query62	755	678	655	655
query63	209	185	183	183
query64	1406	634	614	614
query65	3274	3161	3185	3161
query66	675	305	294	294
query67	15942	15557	15551	15551
query68	4593	587	593	587
query69	455	293	294	293
query70	1183	1108	1061	1061
query71	348	267	279	267
query72	6377	4112	4061	4061
query73	768	328	330	328
query74	9514	9038	8997	8997
query75	3327	2659	2667	2659
query76	1792	910	924	910
query77	473	299	305	299
query78	9927	9262	9237	9237
query79	1154	563	536	536
query80	895	454	438	438
query81	536	243	242	242
query82	1291	151	142	142
query83	263	141	142	141
query84	278	83	72	72
query85	898	289	275	275
query86	338	307	294	294
query87	4560	4250	4343	4250
query88	3168	2340	2346	2340
query89	407	280	285	280
query90	1923	190	187	187
query91	180	142	139	139
query92	67	47	49	47
query93	1689	531	529	529
query94	733	298	282	282
query95	339	253	262	253
query96	610	278	279	278
query97	3230	3158	3108	3108
query98	221	207	198	198
query99	1664	1308	1285	1285
Total cold run time: 314493 ms
Total hot run time: 195840 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.46 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 956178e05db2df0c4213a72f27b1789abfe2100e, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.65	0.10	0.11
query5	0.51	0.50	0.49
query6	1.13	0.74	0.71
query7	0.02	0.02	0.01
query8	0.04	0.03	0.02
query9	0.57	0.50	0.49
query10	0.55	0.56	0.55
query11	0.14	0.11	0.10
query12	0.15	0.11	0.12
query13	0.61	0.59	0.60
query14	2.96	2.99	3.11
query15	0.91	0.84	0.81
query16	0.39	0.38	0.39
query17	1.04	1.07	1.06
query18	0.19	0.20	0.19
query19	1.97	1.89	1.96
query20	0.01	0.00	0.01
query21	15.36	0.57	0.58
query22	2.56	1.66	2.46
query23	17.35	0.89	0.73
query24	2.42	1.01	1.20
query25	0.23	0.14	0.04
query26	0.61	0.14	0.13
query27	0.04	0.04	0.04
query28	10.94	1.09	1.07
query29	12.53	3.22	3.18
query30	0.25	0.06	0.06
query31	2.87	0.40	0.37
query32	3.27	0.47	0.46
query33	2.99	3.07	2.97
query34	17.07	4.33	4.38
query35	4.40	4.44	4.43
query36	0.64	0.49	0.50
query37	0.08	0.05	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.15	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.17 s
Total hot run time: 32.46 s

@Toms1999
Copy link
Contributor Author

run buildall

# Conflicts:
#	regression-test/suites/load_p0/routine_load/test_show_routine_load_expr.groovy
@Toms1999
Copy link
Contributor Author

run buildall

@Toms1999
Copy link
Contributor Author

run p0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants