[Feat](Nereids) support unset command #43103

LiBinfeng-01 · 2024-11-01T13:09:23Z

What problem does this PR solve?

Issue Number: close #43102

Problem Summary:

Check List (For Committer)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No colde files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.
Release note

None

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-01T13:09:27Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

LiBinfeng-01 · 2024-11-01T13:10:15Z

run buildall

doris-robot · 2024-11-01T14:20:47Z

TPC-H: Total hot run time: 41506 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a1babf84fde311041e691e61920b99eeaea6bbbb, data reload: false

------ Round 1 ----------------------------------
q1	17629	7446	7302	7302
q2	2053	166	175	166
q3	10558	1091	1177	1091
q4	10572	877	816	816
q5	7748	3101	3087	3087
q6	234	141	144	141
q7	1030	606	611	606
q8	9359	1928	2022	1928
q9	6616	6479	6479	6479
q10	7047	2443	2433	2433
q11	466	259	253	253
q12	414	218	213	213
q13	17787	3025	3026	3025
q14	255	223	209	209
q15	576	524	514	514
q16	647	597	581	581
q17	993	555	532	532
q18	7195	6678	6719	6678
q19	1337	1003	1026	1003
q20	456	180	184	180
q21	3963	3265	3267	3265
q22	1151	1044	1004	1004
Total cold run time: 108086 ms
Total hot run time: 41506 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7255	7561	7251	7251
q2	326	227	228	227
q3	3019	2960	3009	2960
q4	2096	1875	1785	1785
q5	5720	5809	5796	5796
q6	229	141	138	138
q7	2285	1844	1793	1793
q8	3387	3528	3485	3485
q9	8967	8986	8891	8891
q10	3616	3587	3594	3587
q11	614	509	519	509
q12	850	638	650	638
q13	10295	3249	3193	3193
q14	319	275	279	275
q15	582	511	527	511
q16	698	638	624	624
q17	1871	1644	1627	1627
q18	8397	7781	7624	7624
q19	1750	1543	1539	1539
q20	2078	1880	1880	1880
q21	5508	5347	5608	5347
q22	1148	1075	1067	1067
Total cold run time: 71010 ms
Total hot run time: 60747 ms

doris-robot · 2024-11-01T14:33:22Z

TPC-DS: Total hot run time: 195886 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a1babf84fde311041e691e61920b99eeaea6bbbb, data reload: false

query1	1232	925	912	912
query2	6237	2246	2126	2126
query3	10939	4321	4021	4021
query4	68409	29163	23891	23891
query5	4957	443	419	419
query6	399	174	168	168
query7	5625	287	303	287
query8	319	226	222	222
query9	9341	2755	2748	2748
query10	461	254	245	245
query11	17483	15344	15839	15344
query12	152	105	102	102
query13	1562	440	417	417
query14	10814	7537	6412	6412
query15	198	176	169	169
query16	7040	467	472	467
query17	993	552	544	544
query18	1777	295	288	288
query19	202	149	152	149
query20	116	107	109	107
query21	207	99	102	99
query22	4629	4133	4217	4133
query23	34541	34231	34090	34090
query24	5995	2806	2741	2741
query25	513	393	391	391
query26	644	155	152	152
query27	1707	284	284	284
query28	4009	2441	2429	2429
query29	678	418	416	416
query30	233	161	152	152
query31	979	794	789	789
query32	67	56	55	55
query33	448	267	273	267
query34	913	519	512	512
query35	856	728	734	728
query36	1080	950	935	935
query37	123	76	74	74
query38	4334	4186	4177	4177
query39	1465	1413	1433	1413
query40	193	96	96	96
query41	47	45	48	45
query42	106	97	97	97
query43	546	521	520	520
query44	1170	818	817	817
query45	186	160	175	160
query46	1130	710	706	706
query47	1929	1861	1892	1861
query48	422	327	326	326
query49	741	408	389	389
query50	814	403	389	389
query51	7372	7169	7089	7089
query52	101	92	88	88
query53	261	179	182	179
query54	524	405	407	405
query55	78	77	77	77
query56	264	252	276	252
query57	1312	1162	1180	1162
query58	210	226	198	198
query59	3218	3296	3235	3235
query60	265	246	249	246
query61	102	95	105	95
query62	778	686	700	686
query63	212	186	186	186
query64	1314	603	613	603
query65	3300	3244	3228	3228
query66	695	305	299	299
query67	16049	15797	15667	15667
query68	3193	585	592	585
query69	409	247	249	247
query70	1176	1161	1101	1101
query71	341	256	248	248
query72	6071	4011	3954	3954
query73	749	360	352	352
query74	10184	9192	9007	9007
query75	3381	2653	2668	2653
query76	1678	1034	1030	1030
query77	468	275	267	267
query78	10407	9414	9377	9377
query79	1387	580	590	580
query80	843	428	440	428
query81	527	238	244	238
query82	1303	116	113	113
query83	187	133	133	133
query84	275	69	68	68
query85	838	285	278	278
query86	344	313	307	307
query87	4837	4679	4703	4679
query88	3366	2186	2149	2149
query89	415	284	295	284
query90	2001	179	192	179
query91	132	109	103	103
query92	59	48	47	47
query93	1556	536	528	528
query94	771	288	288	288
query95	350	238	244	238
query96	613	274	276	274
query97	2860	2719	2696	2696
query98	221	205	195	195
query99	1556	1322	1284	1284
Total cold run time: 317514 ms
Total hot run time: 195886 ms

doris-robot · 2024-11-01T14:38:51Z

ClickBench: Total hot run time: 32.04 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a1babf84fde311041e691e61920b99eeaea6bbbb, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.03
query3	0.22	0.06	0.06
query4	1.65	0.10	0.09
query5	0.41	0.40	0.41
query6	1.16	0.66	0.64
query7	0.01	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.56	0.55	0.57
query11	0.14	0.10	0.11
query12	0.14	0.11	0.12
query13	0.61	0.59	0.59
query14	2.84	2.73	2.72
query15	0.90	0.82	0.83
query16	0.39	0.37	0.38
query17	1.04	1.04	1.04
query18	0.20	0.20	0.20
query19	1.83	1.78	1.93
query20	0.02	0.01	0.01
query21	15.37	0.59	0.57
query22	2.76	2.09	1.45
query23	17.11	0.91	0.86
query24	2.74	1.14	0.76
query25	0.15	0.18	0.08
query26	0.55	0.15	0.14
query27	0.04	0.04	0.04
query28	10.92	1.10	1.07
query29	12.54	3.24	3.24
query30	0.25	0.06	0.05
query31	2.89	0.39	0.37
query32	3.27	0.47	0.46
query33	3.00	3.04	3.02
query34	16.84	4.52	4.50
query35	4.57	4.58	4.58
query36	0.66	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.03
query40	0.16	0.12	0.11
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.95 s
Total hot run time: 32.04 s

LiBinfeng-01 · 2024-11-01T14:50:58Z

run p0

LiBinfeng-01 · 2024-11-02T03:06:43Z

run p0

englefly · 2024-11-04T02:30:41Z

...main/java/org/apache/doris/nereids/trees/plans/commands/UnsetDefaultStorageVaultCommand.java

+            throw new AnalysisException("Storage Vault is only supported for cloud mode");
+        }
+        if (!FeConstants.runningUnitTest) {
+            // In legacy cloud mode, some s3 back-ended storage does need to use storage vault.


vault => value?

storage vault is a word group

englefly · 2024-11-04T02:32:08Z

...e-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/UnsetVariableCommand.java

+/**
+ * UnSetVarOp
+ */
+public class UnsetVariableCommand extends Command implements Forward {


Forward or ForwardWithSync?

Forward, because it should be decided by toRedirectStatus

englefly · 2024-11-04T02:33:19Z

...e-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/UnsetVariableCommand.java

+
+    private SetType setType;
+
+    // variables to restore


why plural？does this command unset one variable or a set of “variables”？

only one variable can be unset at one time, nice check

...e-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/UnsetVariableCommand.java

LiBinfeng-01 · 2024-11-04T08:01:10Z

run buildall

LiBinfeng-01 · 2024-11-04T08:27:51Z

run buildall

doris-robot · 2024-11-04T08:47:39Z

TPC-H: Total hot run time: 41245 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ef709858d1c9552c980b426f04aff6ff159686e9, data reload: false

------ Round 1 ----------------------------------
q1	17564	7420	7280	7280
q2	2062	168	164	164
q3	10817	1093	1150	1093
q4	10570	884	903	884
q5	7731	3051	3085	3051
q6	237	152	153	152
q7	1034	609	602	602
q8	9356	1955	2017	1955
q9	6573	6422	6445	6422
q10	7062	2397	2449	2397
q11	457	250	261	250
q12	417	223	220	220
q13	17792	3003	3007	3003
q14	262	215	209	209
q15	562	521	517	517
q16	641	578	582	578
q17	972	581	580	580
q18	7257	6598	6706	6598
q19	1356	1092	997	997
q20	513	182	195	182
q21	4001	3100	3084	3084
q22	1117	1027	1034	1027
Total cold run time: 108353 ms
Total hot run time: 41245 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7348	7282	7253	7253
q2	318	221	227	221
q3	2997	2954	2919	2919
q4	2091	1880	1821	1821
q5	5733	5725	5783	5725
q6	226	144	145	144
q7	2269	1857	1775	1775
q8	3343	3575	3431	3431
q9	8890	8920	8890	8890
q10	3596	3540	3543	3540
q11	605	519	501	501
q12	864	641	642	641
q13	10372	3149	3274	3149
q14	309	270	277	270
q15	565	531	534	531
q16	675	628	646	628
q17	1812	1647	1650	1647
q18	8195	7838	7739	7739
q19	1723	1555	1658	1555
q20	2077	1862	1869	1862
q21	5507	5520	5535	5520
q22	1150	1102	1046	1046
Total cold run time: 70665 ms
Total hot run time: 60808 ms

doris-robot · 2024-11-04T09:00:12Z

TPC-DS: Total hot run time: 196305 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ef709858d1c9552c980b426f04aff6ff159686e9, data reload: false

query1	1193	897	921	897
query2	6248	2118	2080	2080
query3	10788	4014	4103	4014
query4	67761	28885	23645	23645
query5	4898	460	443	443
query6	405	186	183	183
query7	5521	292	288	288
query8	311	228	225	225
query9	8693	2684	2645	2645
query10	445	258	247	247
query11	17299	15329	15821	15329
query12	156	104	98	98
query13	1487	433	407	407
query14	9842	7359	7620	7359
query15	209	192	192	192
query16	7053	450	461	450
query17	1039	557	560	557
query18	1802	326	293	293
query19	198	150	153	150
query20	116	116	114	114
query21	203	102	99	99
query22	4627	4256	4406	4256
query23	35495	34329	34214	34214
query24	6000	2764	2727	2727
query25	522	394	399	394
query26	652	159	157	157
query27	1658	295	283	283
query28	4063	2432	2393	2393
query29	677	434	426	426
query30	234	153	155	153
query31	984	785	817	785
query32	65	55	56	55
query33	432	268	268	268
query34	905	502	508	502
query35	853	727	754	727
query36	1083	944	963	944
query37	125	74	71	71
query38	4362	4282	4246	4246
query39	1497	1442	1431	1431
query40	199	101	101	101
query41	48	46	46	46
query42	105	95	95	95
query43	539	488	506	488
query44	1167	803	810	803
query45	179	167	169	167
query46	1119	705	720	705
query47	1952	1819	1858	1819
query48	405	315	319	315
query49	734	420	391	391
query50	807	380	390	380
query51	7206	7204	7113	7113
query52	102	86	87	86
query53	254	175	177	175
query54	504	391	383	383
query55	73	73	76	73
query56	246	234	237	234
query57	1272	1197	1147	1147
query58	214	205	202	202
query59	3259	3081	3050	3050
query60	273	245	242	242
query61	103	97	98	97
query62	808	684	677	677
query63	211	184	187	184
query64	1331	655	617	617
query65	3255	3184	3239	3184
query66	696	309	303	303
query67	16148	15637	15723	15637
query68	3964	573	557	557
query69	439	252	260	252
query70	1191	1120	1128	1120
query71	354	251	244	244
query72	6130	4073	4008	4008
query73	753	360	355	355
query74	10138	8974	9073	8974
query75	3307	2660	2689	2660
query76	1837	1012	985	985
query77	444	265	269	265
query78	10357	9433	9440	9433
query79	1495	584	593	584
query80	1039	427	427	427
query81	521	237	243	237
query82	458	116	109	109
query83	159	134	140	134
query84	277	66	66	66
query85	916	288	296	288
query86	394	306	312	306
query87	4782	4753	4611	4611
query88	3621	2190	2152	2152
query89	405	283	293	283
query90	1964	187	184	184
query91	138	99	99	99
query92	64	48	46	46
query93	1663	547	529	529
query94	792	258	293	258
query95	351	241	243	241
query96	623	276	289	276
query97	2924	2749	2687	2687
query98	214	202	192	192
query99	1561	1316	1282	1282
Total cold run time: 316385 ms
Total hot run time: 196305 ms

doris-robot · 2024-11-04T09:05:42Z

ClickBench: Total hot run time: 32.69 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ef709858d1c9552c980b426f04aff6ff159686e9, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.22	0.07	0.07
query4	1.64	0.10	0.10
query5	0.43	0.42	0.42
query6	1.13	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.56	0.49	0.50
query10	0.55	0.57	0.55
query11	0.16	0.10	0.10
query12	0.13	0.12	0.10
query13	0.61	0.59	0.59
query14	2.82	2.79	2.73
query15	0.90	0.83	0.83
query16	0.38	0.38	0.39
query17	1.01	1.07	1.06
query18	0.20	0.20	0.21
query19	1.98	1.91	2.04
query20	0.02	0.01	0.01
query21	15.37	0.59	0.56
query22	2.41	2.44	1.81
query23	16.93	1.05	0.88
query24	2.51	1.14	0.99
query25	0.22	0.12	0.18
query26	0.38	0.13	0.15
query27	0.06	0.05	0.04
query28	11.10	1.09	1.07
query29	12.55	3.22	3.22
query30	0.25	0.06	0.06
query31	2.88	0.39	0.38
query32	3.27	0.46	0.44
query33	2.99	3.03	3.00
query34	17.00	4.49	4.47
query35	4.52	4.44	4.49
query36	0.66	0.48	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.03
query40	0.15	0.12	0.13
query41	0.07	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.48 s
Total hot run time: 32.69 s

github-actions · 2024-11-04T11:11:04Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-04T11:11:06Z

PR approved by anyone and no changes requested.

…ic tag (#44530) Related PR: #43103 Problem Summary: When getting profile of old planner, we would find that it can not get profile for some statements, which is not expected to check results of nereids planner also revert show trash command, cause of implementation problem

…dent (#44569) Related PR: #43103 When set and unset commands set the same variable, it would failed by accident duplicate case added by related pr

[Feat](Nereids) support unset command

a1babf8

englefly reviewed Nov 4, 2024

View reviewed changes

fix comments

77ec0e0

LiBinfeng-01 force-pushed the add_unset_variable branch from 8b015d0 to 77ec0e0 Compare November 4, 2024 07:58

rename test files

ef70985

englefly approved these changes Nov 4, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 4, 2024

github-actions bot added the reviewed label Nov 4, 2024

starocean999 approved these changes Nov 4, 2024

View reviewed changes

starocean999 merged commit ee635a8 into apache:master Nov 4, 2024
26 of 28 checks passed

This was referenced Nov 25, 2024

[fix](regression) add exception when can not find profile with specific tag #44530

Merged

[fix](cases) fix regression test set and unset command failed by accident #44569

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feat](Nereids) support unset command #43103

[Feat](Nereids) support unset command #43103

LiBinfeng-01 commented Nov 1, 2024 •

edited

Loading

doris-robot commented Nov 1, 2024

LiBinfeng-01 commented Nov 1, 2024

doris-robot commented Nov 1, 2024

doris-robot commented Nov 1, 2024

doris-robot commented Nov 1, 2024

LiBinfeng-01 commented Nov 1, 2024

LiBinfeng-01 commented Nov 2, 2024

englefly Nov 4, 2024

LiBinfeng-01 Nov 4, 2024

englefly Nov 4, 2024

LiBinfeng-01 Nov 4, 2024

englefly Nov 4, 2024

LiBinfeng-01 Nov 4, 2024

LiBinfeng-01 commented Nov 4, 2024

LiBinfeng-01 commented Nov 4, 2024

doris-robot commented Nov 4, 2024

doris-robot commented Nov 4, 2024

doris-robot commented Nov 4, 2024

github-actions bot commented Nov 4, 2024

github-actions bot commented Nov 4, 2024

[Feat](Nereids) support unset command #43103

[Feat](Nereids) support unset command #43103

Conversation

LiBinfeng-01 commented Nov 1, 2024 • edited Loading

What problem does this PR solve?

Check List (For Committer)

Check List (For Reviewer who merge this PR)

doris-robot commented Nov 1, 2024

LiBinfeng-01 commented Nov 1, 2024

doris-robot commented Nov 1, 2024

doris-robot commented Nov 1, 2024

doris-robot commented Nov 1, 2024

LiBinfeng-01 commented Nov 1, 2024

LiBinfeng-01 commented Nov 2, 2024

englefly Nov 4, 2024

Choose a reason for hiding this comment

LiBinfeng-01 Nov 4, 2024

Choose a reason for hiding this comment

englefly Nov 4, 2024

Choose a reason for hiding this comment

LiBinfeng-01 Nov 4, 2024

Choose a reason for hiding this comment

englefly Nov 4, 2024

Choose a reason for hiding this comment

LiBinfeng-01 Nov 4, 2024

Choose a reason for hiding this comment

LiBinfeng-01 commented Nov 4, 2024

LiBinfeng-01 commented Nov 4, 2024

doris-robot commented Nov 4, 2024

doris-robot commented Nov 4, 2024

doris-robot commented Nov 4, 2024

github-actions bot commented Nov 4, 2024

github-actions bot commented Nov 4, 2024

LiBinfeng-01 commented Nov 1, 2024 •

edited

Loading