Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[refine](rf) Adjust the calculation logic for rf's always_true. #41477

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[refine](rf) Adjust the calculation logic for rf's always_true. #41477

wants to merge 2 commits into from
+101 −60

Conversation

Mryange
Copy link
Contributor

@Mryange Mryange commented Sep 29, 2024

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Mryange
Copy link
Contributor Author

Mryange commented Sep 29, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.29% (9627/25819)
Line Coverage: 28.69% (79704/277772)
Region Coverage: 28.12% (41209/146532)
Branch Coverage: 24.74% (20980/84812)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3e49928daa7797a8b4e0be2b5d1d7d5c4135a595_3e49928daa7797a8b4e0be2b5d1d7d5c4135a595/report/index.html

@Mryange
Copy link
Contributor Author

Mryange commented Sep 29, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40965 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3e49928daa7797a8b4e0be2b5d1d7d5c4135a595, data reload: false

------ Round 1 ----------------------------------
q1	18140	7410	7315	7315
q2	2766	159	156	156
q3	11693	1132	1204	1132
q4	10773	875	869	869
q5	8069	3081	3061	3061
q6	239	150	147	147
q7	1062	634	608	608
q8	9343	1922	1934	1922
q9	6549	6427	6399	6399
q10	7034	2427	2396	2396
q11	436	234	235	234
q12	399	212	218	212
q13	17766	2969	2979	2969
q14	237	211	203	203
q15	576	508	524	508
q16	645	604	581	581
q17	982	578	540	540
q18	7219	6678	6620	6620
q19	1353	1027	920	920
q20	478	189	182	182
q21	3960	3057	2999	2999
q22	1095	1015	992	992
Total cold run time: 110814 ms
Total hot run time: 40965 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7207	7190	7250	7190
q2	334	233	225	225
q3	2905	2764	2823	2764
q4	1950	1702	1727	1702
q5	5411	5490	5445	5445
q6	227	136	139	136
q7	2200	1737	1708	1708
q8	3212	3402	3458	3402
q9	8583	8475	8544	8475
q10	3446	3413	3416	3413
q11	586	473	470	470
q12	758	568	596	568
q13	5894	2959	2978	2959
q14	288	265	269	265
q15	550	496	519	496
q16	687	622	623	622
q17	1784	1569	1554	1554
q18	7721	7452	7459	7452
q19	1652	1559	1561	1559
q20	2052	1774	1841	1774
q21	5374	5206	5067	5067
q22	1078	1050	1002	1002
Total cold run time: 63899 ms
Total hot run time: 58248 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.29% (9628/25819)
Line Coverage: 28.68% (79668/277783)
Region Coverage: 28.12% (41198/146532)
Branch Coverage: 24.73% (20977/84812)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3e49928daa7797a8b4e0be2b5d1d7d5c4135a595_3e49928daa7797a8b4e0be2b5d1d7d5c4135a595/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 192403 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3e49928daa7797a8b4e0be2b5d1d7d5c4135a595, data reload: false

query1	980	365	357	357
query2	6509	2108	2098	2098
query3	6780	220	231	220
query4	33992	23468	23418	23418
query5	4317	462	450	450
query6	270	183	160	160
query7	4596	287	293	287
query8	274	223	221	221
query9	9660	2648	2642	2642
query10	478	283	277	277
query11	17923	15188	15291	15188
query12	148	97	99	97
query13	1675	428	406	406
query14	10778	7539	7379	7379
query15	256	171	177	171
query16	7990	480	483	480
query17	1669	586	575	575
query18	2006	295	299	295
query19	361	157	155	155
query20	121	105	107	105
query21	219	104	101	101
query22	4711	4423	4464	4423
query23	34859	34177	34019	34019
query24	11141	2761	2776	2761
query25	636	396	409	396
query26	1175	158	158	158
query27	2810	279	282	279
query28	8044	2417	2403	2403
query29	844	428	464	428
query30	323	155	154	154
query31	1031	820	820	820
query32	101	57	55	55
query33	775	285	317	285
query34	950	511	503	503
query35	947	748	746	746
query36	1115	967	980	967
query37	150	87	89	87
query38	4062	3873	3908	3873
query39	1483	1425	1444	1425
query40	201	96	94	94
query41	46	42	43	42
query42	122	96	95	95
query43	538	496	479	479
query44	1247	784	792	784
query45	196	164	169	164
query46	1133	698	703	698
query47	1987	1812	1838	1812
query48	400	315	319	315
query49	1122	417	395	395
query50	809	378	381	378
query51	7099	7003	6993	6993
query52	99	86	87	86
query53	261	187	187	187
query54	1314	409	420	409
query55	90	77	78	77
query56	253	263	250	250
query57	1276	1165	1142	1142
query58	230	232	242	232
query59	3160	3004	2917	2917
query60	286	264	254	254
query61	104	108	105	105
query62	874	663	654	654
query63	230	192	185	185
query64	5325	642	612	612
query65	3249	3195	3181	3181
query66	1403	316	297	297
query67	16008	15567	15642	15567
query68	3506	575	549	549
query69	408	284	289	284
query70	1157	1075	1114	1075
query71	326	267	266	266
query72	6395	5061	5008	5008
query73	769	355	353	353
query74	9947	9004	9016	9004
query75	3391	2689	2649	2649
query76	2194	911	929	911
query77	409	289	303	289
query78	10721	9614	9691	9614
query79	1807	590	601	590
query80	1188	485	436	436
query81	569	238	246	238
query82	916	131	140	131
query83	242	143	138	138
query84	234	69	72	69
query85	1281	288	288	288
query86	437	296	292	292
query87	4436	4239	4402	4239
query88	2922	2171	2141	2141
query89	399	284	285	284
query90	1915	191	188	188
query91	140	103	103	103
query92	71	52	45	45
query93	1383	542	533	533
query94	953	276	294	276
query95	349	245	245	245
query96	617	281	280	280
query97	3216	3136	3170	3136
query98	212	209	189	189
query99	1522	1292	1301	1292
Total cold run time: 299253 ms
Total hot run time: 192403 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.21 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3e49928daa7797a8b4e0be2b5d1d7d5c4135a595, data reload: false

query1	0.05	0.04	0.04
query2	0.06	0.03	0.02
query3	0.22	0.06	0.06
query4	1.65	0.10	0.10
query5	0.51	0.51	0.50
query6	1.14	0.72	0.71
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.57	0.58	0.55
query11	0.13	0.10	0.10
query12	0.13	0.10	0.11
query13	0.62	0.59	0.59
query14	2.69	2.75	2.86
query15	0.88	0.82	0.82
query16	0.40	0.38	0.39
query17	1.05	1.09	1.05
query18	0.23	0.23	0.22
query19	1.98	1.83	2.01
query20	0.01	0.01	0.01
query21	15.36	0.61	0.60
query22	3.13	2.37	2.08
query23	16.74	0.89	0.87
query24	3.04	1.23	1.28
query25	0.34	0.06	0.07
query26	0.49	0.14	0.13
query27	0.05	0.04	0.03
query28	10.30	1.10	1.06
query29	12.57	3.21	3.20
query30	0.25	0.06	0.05
query31	2.88	0.39	0.36
query32	3.28	0.46	0.46
query33	3.02	3.03	3.01
query34	17.03	4.46	4.47
query35	4.49	4.48	4.51
query36	0.70	0.48	0.48
query37	0.09	0.05	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.15	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 107.08 s
Total hot run time: 33.21 s

HappenLee
HappenLee reviewed Sep 30, 2024
View reviewed changes
be/src/vec/exprs/vruntimefilter_wrapper.h Outdated
@@ -72,34 +72,58 @@ class VRuntimeFilterWrapper final : public VExpr {
_always_true_counter = always_true_counter;
}

template <typename T, typename TT>
void update_counters(int64_t filter_rows, int64_t input_rows) {
if (_expr_filtered_rows_counter) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what cast the counter is nullptr

@Mryange
Copy link
Contributor Author

Mryange commented Oct 7, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41471 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d86c1b5465a9d368b74c33e7ef00a0343bd1982e, data reload: false

------ Round 1 ----------------------------------
q1	17578	7475	7222	7222
q2	2029	295	266	266
q3	11586	1087	1144	1087
q4	10594	805	808	805
q5	7752	3112	3043	3043
q6	234	150	148	148
q7	1107	654	611	611
q8	9340	1916	1949	1916
q9	6584	6431	6415	6415
q10	7017	2428	2476	2428
q11	446	247	245	245
q12	423	220	211	211
q13	17842	2979	3001	2979
q14	254	217	210	210
q15	581	516	527	516
q16	637	591	590	590
q17	983	547	512	512
q18	7274	6798	6806	6798
q19	1353	1026	1065	1026
q20	481	190	183	183
q21	4010	3276	3322	3276
q22	1074	1005	984	984
Total cold run time: 109179 ms
Total hot run time: 41471 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7265	7232	7197	7197
q2	335	230	231	230
q3	3093	2947	2989	2947
q4	2104	1872	1782	1782
q5	5785	5792	5789	5789
q6	230	142	144	142
q7	2276	1858	1849	1849
q8	3433	3488	3495	3488
q9	8978	8922	8904	8904
q10	3632	3604	3558	3558
q11	590	504	478	478
q12	845	638	600	600
q13	9912	3224	3171	3171
q14	312	285	277	277
q15	579	515	518	515
q16	711	665	643	643
q17	1838	1605	1597	1597
q18	8296	7952	7654	7654
q19	1712	1596	1529	1529
q20	2100	1887	1885	1885
q21	5498	5428	5419	5419
q22	1167	1030	1074	1030
Total cold run time: 70691 ms
Total hot run time: 60684 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.32% (9632/25809)
Line Coverage: 28.71% (79897/278296)
Region Coverage: 28.14% (41302/146755)
Branch Coverage: 24.76% (21042/84972)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d86c1b5465a9d368b74c33e7ef00a0343bd1982e_d86c1b5465a9d368b74c33e7ef00a0343bd1982e/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 193117 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d86c1b5465a9d368b74c33e7ef00a0343bd1982e, data reload: false

query1	932	400	407	400
query2	6253	2125	2060	2060
query3	8685	194	204	194
query4	33665	23727	23621	23621
query5	3720	461	449	449
query6	280	163	159	159
query7	4188	287	285	285
query8	274	234	233	233
query9	9273	2692	2695	2692
query10	464	273	266	266
query11	17741	15309	15345	15309
query12	148	103	96	96
query13	1589	469	420	420
query14	9845	7598	7471	7471
query15	257	168	183	168
query16	7902	431	470	431
query17	1672	611	567	567
query18	1949	316	327	316
query19	364	161	159	159
query20	134	115	113	113
query21	221	106	109	106
query22	4943	4408	4527	4408
query23	34962	34165	34129	34129
query24	10954	2738	2764	2738
query25	608	402	406	402
query26	1124	155	163	155
query27	2336	291	282	282
query28	7521	2454	2435	2435
query29	810	425	421	421
query30	254	160	154	154
query31	1039	771	818	771
query32	101	54	55	54
query33	775	298	280	280
query34	915	515	505	505
query35	890	736	731	731
query36	1105	957	962	957
query37	146	84	89	84
query38	4038	3933	3916	3916
query39	1483	1459	1433	1433
query40	212	98	101	98
query41	47	44	45	44
query42	117	97	96	96
query43	544	493	492	492
query44	1379	807	812	807
query45	202	165	172	165
query46	1144	702	689	689
query47	1928	1827	1861	1827
query48	408	336	329	329
query49	892	403	400	400
query50	810	390	376	376
query51	7043	7051	6894	6894
query52	97	89	89	89
query53	255	179	181	179
query54	1192	414	410	410
query55	79	79	79	79
query56	291	264	284	264
query57	1249	1137	1146	1137
query58	229	225	245	225
query59	3262	3130	3056	3056
query60	290	265	262	262
query61	107	99	102	99
query62	861	671	655	655
query63	219	185	181	181
query64	4035	628	609	609
query65	3260	3209	3184	3184
query66	840	304	297	297
query67	16066	15687	15763	15687
query68	4359	560	576	560
query69	487	289	314	289
query70	1184	1147	1126	1126
query71	357	257	268	257
query72	7577	4999	4986	4986
query73	776	358	366	358
query74	10174	8982	9013	8982
query75	3420	2722	2653	2653
query76	2948	916	930	916
query77	600	308	289	289
query78	10459	9621	9513	9513
query79	1972	583	601	583
query80	2480	459	440	440
query81	588	239	249	239
query82	701	134	140	134
query83	292	136	136	136
query84	271	74	68	68
query85	1586	289	296	289
query86	426	269	300	269
query87	4485	4307	4278	4278
query88	3206	2187	2148	2148
query89	401	283	298	283
query90	2045	187	186	186
query91	142	100	109	100
query92	60	48	48	48
query93	1924	562	542	542
query94	1153	286	288	286
query95	348	242	240	240
query96	616	280	281	280
query97	3300	3169	3196	3169
query98	213	208	205	205
query99	1543	1316	1265	1265
Total cold run time: 300711 ms
Total hot run time: 193117 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.74 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d86c1b5465a9d368b74c33e7ef00a0343bd1982e, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.03	0.03
query3	0.22	0.06	0.06
query4	1.64	0.10	0.10
query5	0.53	0.52	0.52
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.56	0.51	0.49
query10	0.55	0.56	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.59	0.59
query14	2.71	2.72	2.73
query15	0.91	0.83	0.82
query16	0.38	0.39	0.37
query17	1.04	1.07	1.02
query18	0.20	0.20	0.20
query19	1.88	1.85	1.98
query20	0.01	0.01	0.01
query21	15.35	0.62	0.58
query22	2.31	2.33	2.44
query23	17.15	1.02	0.84
query24	2.86	0.50	1.28
query25	0.23	0.07	0.05
query26	0.49	0.14	0.14
query27	0.05	0.04	0.04
query28	11.04	1.09	1.07
query29	12.56	3.23	3.23
query30	0.25	0.06	0.07
query31	2.87	0.38	0.38
query32	3.30	0.47	0.47
query33	3.00	2.99	3.00
query34	16.91	4.44	4.49
query35	4.48	4.50	4.49
query36	0.65	0.51	0.48
query37	0.08	0.07	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.77 s
Total hot run time: 32.74 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HappenLee HappenLee HappenLee left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Mryange @doris-robot @HappenLee