Skip to content

Conversation

@liutang123
Copy link
Contributor

What problem does this PR solve?

When client send a prepare command to non master frontend like INSERT INTO xxxx VALUES(?) and then execute it whith some params.
The non master frontend will forward the origin stmt to master and analyze error will occur.
In this PR, we forward the origin sql and param to master. Master prepare a logic plan with place holder and then use the param and logic plan to generate physical plan.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 5, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow
Copy link
Contributor

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.15% (1063/1294)
Line Coverage: 65.69% (17630/26840)
Region Coverage: 65.13% (8688/13340)
Branch Coverage: 55.09% (4688/8510)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bf60474fd12c9a43fc761b06fe7f8a69d73e012b_bf60474fd12c9a43fc761b06fe7f8a69d73e012b_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 32623 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bf60474fd12c9a43fc761b06fe7f8a69d73e012b, data reload: false

------ Round 1 ----------------------------------
q1	17578	5170	5131	5131
q2	2060	317	183	183
q3	10378	1295	711	711
q4	10211	1045	537	537
q5	8040	2414	2372	2372
q6	201	172	136	136
q7	922	747	612	612
q8	9329	1289	1059	1059
q9	5004	4558	4531	4531
q10	6834	2296	1871	1871
q11	492	269	266	266
q12	352	350	222	222
q13	17776	3669	3078	3078
q14	239	233	214	214
q15	548	511	508	508
q16	630	620	595	595
q17	592	881	342	342
q18	6825	6680	6484	6484
q19	1980	966	555	555
q20	327	322	195	195
q21	2763	2124	2047	2047
q22	1056	1016	974	974
Total cold run time: 104137 ms
Total hot run time: 32623 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5261	5122	5107	5107
q2	252	339	231	231
q3	2209	2709	2283	2283
q4	1476	1839	1418	1418
q5	4273	4159	4183	4159
q6	216	167	126	126
q7	2043	1888	1765	1765
q8	2717	2633	2637	2633
q9	7225	7039	7198	7039
q10	2974	3169	2764	2764
q11	577	526	506	506
q12	727	798	630	630
q13	3509	3871	3311	3311
q14	269	289	268	268
q15	544	508	510	508
q16	659	679	626	626
q17	1162	1610	1358	1358
q18	7803	7584	7404	7404
q19	811	827	831	827
q20	1969	2038	1867	1867
q21	5481	5126	4756	4756
q22	1075	1019	1010	1010
Total cold run time: 53232 ms
Total hot run time: 50596 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185542 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bf60474fd12c9a43fc761b06fe7f8a69d73e012b, data reload: false

query1	997	415	390	390
query2	6522	1839	1851	1839
query3	6785	219	209	209
query4	26400	23304	23176	23176
query5	4347	667	481	481
query6	313	206	206	206
query7	4607	504	298	298
query8	308	242	240	240
query9	8593	2554	2574	2554
query10	479	328	252	252
query11	15715	15147	14917	14917
query12	159	115	110	110
query13	1671	525	407	407
query14	9999	6583	6484	6484
query15	207	191	175	175
query16	7440	644	465	465
query17	1176	713	569	569
query18	1813	391	295	295
query19	193	183	150	150
query20	119	121	112	112
query21	205	121	101	101
query22	4100	4234	4116	4116
query23	33975	32947	33139	32947
query24	7741	2361	2395	2361
query25	527	455	394	394
query26	1224	275	163	163
query27	2125	495	326	326
query28	3903	2430	2433	2430
query29	714	560	412	412
query30	280	227	191	191
query31	930	858	753	753
query32	76	64	62	62
query33	558	351	321	321
query34	778	862	507	507
query35	800	835	756	756
query36	972	975	875	875
query37	114	124	74	74
query38	4340	4254	4032	4032
query39	1435	1412	1403	1403
query40	203	113	100	100
query41	53	49	49	49
query42	123	103	102	102
query43	508	505	465	465
query44	1274	778	782	778
query45	178	170	161	161
query46	832	1012	619	619
query47	1745	1788	1728	1728
query48	374	408	306	306
query49	767	526	421	421
query50	683	735	399	399
query51	4191	4151	4146	4146
query52	107	107	90	90
query53	231	252	183	183
query54	499	508	418	418
query55	83	89	79	79
query56	296	269	249	249
query57	1175	1094	1081	1081
query58	244	235	234	234
query59	2452	2655	2747	2655
query60	280	264	257	257
query61	125	119	137	119
query62	807	733	663	663
query63	225	188	195	188
query64	4270	1053	668	668
query65	4387	4295	4329	4295
query66	1092	411	315	315
query67	15643	15332	15560	15332
query68	4767	890	524	524
query69	485	295	274	274
query70	1143	1107	1110	1107
query71	412	280	258	258
query72	5504	3624	3820	3624
query73	739	721	350	350
query74	9047	9127	8960	8960
query75	3134	3220	2692	2692
query76	3196	1184	743	743
query77	455	385	271	271
query78	9890	10220	9301	9301
query79	1238	833	605	605
query80	877	551	443	443
query81	501	268	222	222
query82	411	126	95	95
query83	174	180	166	166
query84	242	98	76	76
query85	755	361	311	311
query86	313	315	287	287
query87	4479	4532	4515	4515
query88	2934	2192	2210	2192
query89	384	327	289	289
query90	1724	201	195	195
query91	142	148	113	113
query92	61	60	62	60
query93	1169	1060	592	592
query94	676	427	297	297
query95	357	270	264	264
query96	495	584	269	269
query97	3295	3344	3297	3297
query98	229	215	201	201
query99	1346	1418	1296	1296
Total cold run time: 266061 ms
Total hot run time: 185542 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.67 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bf60474fd12c9a43fc761b06fe7f8a69d73e012b, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.06
query4	1.62	0.11	0.10
query5	0.58	0.54	0.55
query6	1.23	0.72	0.71
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.52	0.51
query10	0.58	0.58	0.57
query11	0.15	0.10	0.11
query12	0.15	0.11	0.12
query13	0.62	0.59	0.60
query14	2.77	2.74	2.80
query15	0.93	0.85	0.87
query16	0.38	0.37	0.37
query17	1.03	1.04	1.03
query18	0.21	0.19	0.20
query19	1.97	1.79	1.96
query20	0.02	0.01	0.01
query21	15.36	0.87	0.54
query22	0.75	1.21	0.63
query23	14.97	1.40	0.56
query24	6.90	2.09	0.68
query25	0.48	0.20	0.17
query26	0.65	0.16	0.15
query27	0.05	0.05	0.05
query28	9.70	0.85	0.44
query29	12.57	3.96	3.33
query30	0.25	0.09	0.07
query31	2.82	0.60	0.39
query32	3.22	0.56	0.47
query33	3.04	3.03	3.04
query34	15.73	5.16	4.51
query35	4.56	4.50	4.48
query36	0.66	0.49	0.48
query37	0.10	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.05	0.03	0.03
Total cold run time: 105.48 s
Total hot run time: 30.67 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.94% (12278/26727)
Line Coverage 35.38% (103637/292887)
Region Coverage 34.55% (53092/153667)
Branch Coverage 30.24% (26889/88908)

@liutang123
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.15% (1063/1294)
Line Coverage: 65.66% (17624/26840)
Region Coverage: 65.12% (8687/13340)
Branch Coverage: 55.09% (4688/8510)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cd83f9e5942c713c801fc35d0044c09bd160b245_cd83f9e5942c713c801fc35d0044c09bd160b245_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 32943 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cd83f9e5942c713c801fc35d0044c09bd160b245, data reload: false

------ Round 1 ----------------------------------
q1	17611	5636	5118	5118
q2	2051	314	175	175
q3	10383	1302	774	774
q4	10205	1079	532	532
q5	7538	2454	2382	2382
q6	196	171	135	135
q7	910	776	623	623
q8	9312	1372	1146	1146
q9	5036	4871	4937	4871
q10	6835	2345	1909	1909
q11	494	290	266	266
q12	342	353	215	215
q13	17774	3778	3072	3072
q14	241	233	210	210
q15	558	512	497	497
q16	647	610	591	591
q17	597	868	343	343
q18	7076	6636	6406	6406
q19	3746	988	548	548
q20	305	313	189	189
q21	2781	2197	1944	1944
q22	1031	1007	997	997
Total cold run time: 105669 ms
Total hot run time: 32943 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5330	5125	5078	5078
q2	248	338	239	239
q3	2215	2742	2319	2319
q4	1434	1866	1357	1357
q5	4252	4136	4293	4136
q6	219	173	126	126
q7	2047	1967	1828	1828
q8	2670	2612	2528	2528
q9	7316	7228	7216	7216
q10	3049	3197	2754	2754
q11	588	497	472	472
q12	686	772	642	642
q13	3565	3976	3354	3354
q14	294	291	264	264
q15	529	474	482	474
q16	633	719	651	651
q17	1164	1657	1387	1387
q18	7975	7655	7673	7655
q19	849	808	866	808
q20	2025	2141	1882	1882
q21	5576	4798	4820	4798
q22	1063	1039	1039	1039
Total cold run time: 53727 ms
Total hot run time: 51007 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192046 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cd83f9e5942c713c801fc35d0044c09bd160b245, data reload: false

query1	1392	1073	1013	1013
query2	6179	2023	1939	1939
query3	11004	4461	4378	4378
query4	54611	24298	22892	22892
query5	5052	572	500	500
query6	346	205	192	192
query7	4879	522	291	291
query8	316	248	254	248
query9	5740	2563	2587	2563
query10	442	323	253	253
query11	15172	15151	15103	15103
query12	159	115	111	111
query13	1070	538	406	406
query14	10167	7157	6632	6632
query15	219	209	191	191
query16	7161	682	493	493
query17	1090	727	591	591
query18	1586	424	312	312
query19	195	196	174	174
query20	140	118	120	118
query21	215	153	119	119
query22	4439	4566	4412	4412
query23	34358	33543	33363	33363
query24	5880	2410	2418	2410
query25	451	460	407	407
query26	719	274	151	151
query27	1793	508	322	322
query28	3041	2445	2476	2445
query29	589	560	439	439
query30	279	218	194	194
query31	897	888	798	798
query32	75	66	59	59
query33	467	372	301	301
query34	768	871	507	507
query35	812	850	779	779
query36	958	971	906	906
query37	130	97	79	79
query38	4081	4228	4051	4051
query39	1501	1421	1435	1421
query40	210	117	108	108
query41	53	51	50	50
query42	128	110	102	102
query43	534	530	508	508
query44	1303	805	798	798
query45	180	176	174	174
query46	851	1071	676	676
query47	1884	1880	1774	1774
query48	398	415	320	320
query49	708	539	437	437
query50	712	761	443	443
query51	4269	4307	4276	4276
query52	111	100	96	96
query53	232	259	196	196
query54	495	503	434	434
query55	88	85	80	80
query56	279	288	255	255
query57	1177	1194	1112	1112
query58	245	241	243	241
query59	2987	2933	2907	2907
query60	290	282	280	280
query61	127	120	129	120
query62	727	743	724	724
query63	229	186	187	186
query64	1904	1090	715	715
query65	4629	4470	4342	4342
query66	720	384	299	299
query67	15775	15503	15215	15215
query68	6670	871	503	503
query69	564	299	271	271
query70	1220	1140	1095	1095
query71	500	297	262	262
query72	5899	3705	3696	3696
query73	1253	753	344	344
query74	9054	9323	8826	8826
query75	3843	3385	2713	2713
query76	4143	1174	738	738
query77	622	377	324	324
query78	10132	9968	9206	9206
query79	3004	817	576	576
query80	704	527	448	448
query81	484	265	222	222
query82	687	137	95	95
query83	287	168	149	149
query84	286	100	74	74
query85	779	360	352	352
query86	381	303	309	303
query87	4445	4432	4437	4432
query88	3620	2201	2193	2193
query89	409	312	293	293
query90	1799	205	211	205
query91	141	141	109	109
query92	71	62	61	61
query93	2108	1033	569	569
query94	662	402	304	304
query95	348	267	258	258
query96	480	568	269	269
query97	3406	3398	3252	3252
query98	237	200	198	198
query99	1431	1402	1275	1275
Total cold run time: 299287 ms
Total hot run time: 192046 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.41 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cd83f9e5942c713c801fc35d0044c09bd160b245, data reload: false

query1	0.04	0.04	0.03
query2	0.11	0.05	0.05
query3	0.27	0.06	0.05
query4	1.61	0.08	0.07
query5	0.56	0.56	0.54
query6	1.21	0.72	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.62	0.54	0.52
query10	0.58	0.59	0.57
query11	0.26	0.13	0.13
query12	0.25	0.12	0.13
query13	0.63	0.61	0.62
query14	2.72	2.69	2.66
query15	0.99	0.88	0.87
query16	0.37	0.36	0.37
query17	1.03	1.01	1.06
query18	0.19	0.18	0.19
query19	1.96	1.90	1.99
query20	0.02	0.02	0.02
query21	15.34	0.96	0.66
query22	0.94	1.07	0.77
query23	14.71	1.57	0.77
query24	5.36	0.55	0.28
query25	0.17	0.10	0.09
query26	0.56	0.22	0.19
query27	0.09	0.08	0.09
query28	11.01	1.23	0.57
query29	12.52	4.06	3.37
query30	0.28	0.08	0.07
query31	2.80	0.63	0.44
query32	3.22	0.59	0.51
query33	3.11	3.04	3.08
query34	16.71	5.10	4.49
query35	4.48	4.47	4.49
query36	0.64	0.51	0.50
query37	0.20	0.17	0.16
query38	0.17	0.15	0.15
query39	0.05	0.04	0.04
query40	0.20	0.15	0.15
query41	0.11	0.05	0.05
query42	0.06	0.06	0.05
query43	0.06	0.04	0.05
Total cold run time: 106.28 s
Total hot run time: 31.41 s

@liutang123
Copy link
Contributor Author

I test this pr, and it works well in my cluster.
I will submit a PR to branch 2.1.

@liutang123
Copy link
Contributor Author

run beut

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 0.00% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 46.73% (12499/26745)
Line Coverage 36.35% (106530/293039)
Region Coverage 35.42% (54444/153720)
Branch Coverage 30.79% (27383/88936)

Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add test case? example pick none master frontend and send insert/delete/update prepared requests

@liutang123
Copy link
Contributor Author

please add test case? example pick none master frontend and send insert/delete/update prepared requests

The pipeline doris cluster has only one FE.

@eldenmoon
Copy link
Member

run buildall

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
…error (apache#48689)

When client send a prepare command to non master frontend like `INSERT
INTO xxxx VALUES(?)` and then execute it whith some params.
The non master frontend will forward the origin stmt to master and
analyze error will occur.
In this PR, we forward the origin sql and param to master. Master
prepare a logic plan with place holder and then use the param and logic
plan to generate physical plan.
@morrySnow morrySnow added dev/2.1.x dev/3.0.x dev/3.1.x usercase Important user case type label labels Jun 19, 2025
Jibing-Li pushed a commit to Jibing-Li/incubator-doris that referenced this pull request Jun 25, 2025
…error (apache#48689)

When client send a prepare command to non master frontend like `INSERT
INTO xxxx VALUES(?)` and then execute it whith some params.
The non master frontend will forward the origin stmt to master and
analyze error will occur.
In this PR, we forward the origin sql and param to master. Master
prepare a logic plan with place holder and then use the param and logic
plan to generate physical plan.
Jibing-Li pushed a commit to Jibing-Li/incubator-doris that referenced this pull request Jun 25, 2025
…error (apache#48689)

When client send a prepare command to non master frontend like `INSERT
INTO xxxx VALUES(?)` and then execute it whith some params.
The non master frontend will forward the origin stmt to master and
analyze error will occur.
In this PR, we forward the origin sql and param to master. Master
prepare a logic plan with place holder and then use the param and logic
plan to generate physical plan.
Jibing-Li pushed a commit to Jibing-Li/incubator-doris that referenced this pull request Jun 25, 2025
…error (apache#48689)

When client send a prepare command to non master frontend like `INSERT
INTO xxxx VALUES(?)` and then execute it whith some params.
The non master frontend will forward the origin stmt to master and
analyze error will occur.
In this PR, we forward the origin sql and param to master. Master
prepare a logic plan with place holder and then use the param and logic
plan to generate physical plan.
Jibing-Li pushed a commit to Jibing-Li/incubator-doris that referenced this pull request Jun 25, 2025
…error (apache#48689)

When client send a prepare command to non master frontend like `INSERT
INTO xxxx VALUES(?)` and then execute it whith some params.
The non master frontend will forward the origin stmt to master and
analyze error will occur.
In this PR, we forward the origin sql and param to master. Master
prepare a logic plan with place holder and then use the param and logic
plan to generate physical plan.
dataroaring pushed a commit that referenced this pull request Jun 26, 2025
… non master error (#48689) (#52266)

backport: #48689

Co-authored-by: Lijia Liu <liutang123@yeah.net>
morrySnow pushed a commit that referenced this pull request Jun 27, 2025
… non master error #48689 (#52271)

backport: #48689

Co-authored-by: Lijia Liu <liutang123@yeah.net>
yiguolei pushed a commit that referenced this pull request Jun 28, 2025
… non master error (#48689) (#52265)

backport: #48689

Co-authored-by: Lijia Liu <liutang123@yeah.net>
koarz pushed a commit to koarz/doris that referenced this pull request Jul 3, 2025
… non master error (apache#48689) (apache#52266)

backport: apache#48689

Co-authored-by: Lijia Liu <liutang123@yeah.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.11-merged dev/3.0.7-merged dev/3.1.0-merged reviewed usercase Important user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.