Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](binlog) Log table rename when reply #38034

Merged
merged 4 commits into from
Aug 16, 2024

Conversation

lsy3993
Copy link
Contributor

@lsy3993 lsy3993 commented Jul 17, 2024

Proposed changes

Issue Number: close #xxx

Related PR is : #37031 , when replay rename table, addTableRename is needed

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39497 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9b094ffa9df28cc6a1f37059c93200905cdeef50, data reload: false

------ Round 1 ----------------------------------
q1	17651	4865	4291	4291
q2	2015	193	202	193
q3	10499	1193	1055	1055
q4	10202	762	734	734
q5	7533	2691	2641	2641
q6	221	137	138	137
q7	959	599	595	595
q8	9219	2071	2044	2044
q9	8596	6537	6491	6491
q10	8658	3712	3786	3712
q11	472	238	228	228
q12	393	229	222	222
q13	17762	2989	3003	2989
q14	274	237	231	231
q15	503	488	499	488
q16	500	380	383	380
q17	958	624	624	624
q18	8206	7562	7274	7274
q19	6744	1436	1466	1436
q20	734	319	324	319
q21	4971	3128	3213	3128
q22	357	285	286	285
Total cold run time: 117427 ms
Total hot run time: 39497 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4382	4224	4182	4182
q2	366	271	267	267
q3	2981	2849	2892	2849
q4	1996	1701	1700	1700
q5	5651	5501	5498	5498
q6	219	133	137	133
q7	2176	1870	1857	1857
q8	3312	3403	3457	3403
q9	8765	8788	8922	8788
q10	4058	3915	3782	3782
q11	586	507	492	492
q12	833	646	636	636
q13	16436	3196	3168	3168
q14	309	311	276	276
q15	532	495	477	477
q16	493	441	422	422
q17	1789	1512	1510	1510
q18	8077	7996	7919	7919
q19	2044	1605	1518	1518
q20	2158	1868	1859	1859
q21	5062	4849	4642	4642
q22	582	572	502	502
Total cold run time: 72807 ms
Total hot run time: 55880 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174226 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9b094ffa9df28cc6a1f37059c93200905cdeef50, data reload: false

query1	924	374	371	371
query2	6430	1912	1828	1828
query3	6653	206	214	206
query4	28324	17512	17372	17372
query5	3719	488	498	488
query6	248	175	159	159
query7	4569	286	293	286
query8	235	189	185	185
query9	8514	2451	2405	2405
query10	432	283	272	272
query11	11066	10056	10088	10056
query12	114	87	83	83
query13	1624	359	367	359
query14	10355	7737	7565	7565
query15	236	169	167	167
query16	7672	515	468	468
query17	1122	538	525	525
query18	1860	279	276	276
query19	199	148	148	148
query20	90	80	79	79
query21	208	132	129	129
query22	4552	4073	3951	3951
query23	34067	33600	33543	33543
query24	11120	2918	2892	2892
query25	625	395	403	395
query26	707	152	162	152
query27	2247	283	276	276
query28	5900	2097	2094	2094
query29	889	645	621	621
query30	257	154	163	154
query31	982	767	750	750
query32	99	54	58	54
query33	738	373	333	333
query34	901	492	518	492
query35	872	760	753	753
query36	1135	961	963	961
query37	144	90	85	85
query38	2950	2929	2851	2851
query39	893	872	854	854
query40	201	134	126	126
query41	48	46	48	46
query42	112	101	102	101
query43	501	484	458	458
query44	1236	733	731	731
query45	208	164	164	164
query46	1094	720	709	709
query47	1846	1813	1776	1776
query48	372	309	288	288
query49	857	428	425	425
query50	789	396	390	390
query51	6780	6675	6683	6675
query52	100	94	91	91
query53	364	290	287	287
query54	896	458	455	455
query55	77	75	74	74
query56	313	284	283	283
query57	1144	1095	1089	1089
query58	251	259	271	259
query59	2790	2578	2714	2578
query60	330	294	296	294
query61	114	114	114	114
query62	800	655	643	643
query63	319	282	294	282
query64	9265	2330	1747	1747
query65	3157	3130	3118	3118
query66	780	337	348	337
query67	15715	15031	14947	14947
query68	4699	540	535	535
query69	702	425	386	386
query70	1132	1121	1117	1117
query71	470	291	283	283
query72	9163	5676	5653	5653
query73	747	325	324	324
query74	6141	5605	5706	5605
query75	4566	2677	2710	2677
query76	3311	945	896	896
query77	754	314	305	305
query78	9851	9052	10025	9052
query79	9123	530	519	519
query80	1236	499	501	499
query81	586	222	226	222
query82	1447	139	137	137
query83	394	176	170	170
query84	271	83	89	83
query85	1463	316	291	291
query86	441	291	320	291
query87	3344	3086	3065	3065
query88	5113	2411	2431	2411
query89	518	381	403	381
query90	1904	199	200	199
query91	136	109	106	106
query92	66	49	50	49
query93	5886	542	514	514
query94	1289	287	299	287
query95	411	312	325	312
query96	614	272	273	272
query97	3167	3066	3010	3010
query98	215	202	204	202
query99	1539	1248	1289	1248
Total cold run time: 294830 ms
Total hot run time: 174226 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.15 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9b094ffa9df28cc6a1f37059c93200905cdeef50, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.49	0.49
query6	1.14	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.55	0.49	0.49
query10	0.55	0.54	0.55
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.58
query14	0.75	0.80	0.77
query15	0.85	0.82	0.82
query16	0.37	0.35	0.36
query17	1.03	0.98	0.96
query18	0.24	0.23	0.22
query19	1.83	1.71	1.68
query20	0.02	0.01	0.01
query21	15.42	0.73	0.64
query22	3.72	7.03	2.57
query23	18.32	1.32	1.24
query24	2.06	0.26	0.22
query25	0.15	0.10	0.08
query26	0.30	0.22	0.21
query27	0.45	0.23	0.22
query28	13.23	1.01	0.99
query29	12.63	3.28	3.29
query30	0.25	0.06	0.05
query31	2.89	0.39	0.39
query32	3.25	0.47	0.47
query33	2.87	2.95	2.92
query34	16.95	4.34	4.31
query35	4.43	4.45	4.42
query36	0.65	0.48	0.45
query37	0.18	0.15	0.16
query38	0.16	0.16	0.15
query39	0.04	0.04	0.03
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.07	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 109.13 s
Total hot run time: 31.15 s

Copy link
Contributor

@w41ter w41ter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add addTableRename to the replay of OP_RENAME_ROLLUP and OP_RENAME_PARTITION too.

@github-actions github-actions bot added the doing label Aug 5, 2024
@lsy3993
Copy link
Contributor Author

lsy3993 commented Aug 8, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39492 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 042462363ead863f034df700df52c850197cdf39, data reload: false

------ Round 1 ----------------------------------
q1	17625	4419	4285	4285
q2	2013	177	188	177
q3	10484	1218	1137	1137
q4	10150	714	776	714
q5	7510	2502	2532	2502
q6	226	137	139	137
q7	983	593	593	593
q8	9228	1918	1955	1918
q9	8688	6537	6588	6537
q10	7054	2226	2201	2201
q11	446	247	252	247
q12	391	222	223	222
q13	17769	2997	2983	2983
q14	280	244	238	238
q15	540	490	505	490
q16	510	387	390	387
q17	990	700	708	700
q18	8167	7475	7430	7430
q19	5118	1106	1040	1040
q20	684	330	330	330
q21	5440	4216	4649	4216
q22	1112	1008	1028	1008
Total cold run time: 115408 ms
Total hot run time: 39492 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4451	4281	4308	4281
q2	374	263	259	259
q3	2866	2643	2612	2612
q4	1894	1577	1583	1577
q5	5352	5274	5254	5254
q6	220	133	131	131
q7	2040	1655	1690	1655
q8	3172	3331	3347	3331
q9	8428	8369	8414	8369
q10	3357	3195	3173	3173
q11	605	489	511	489
q12	760	586	605	586
q13	17605	2979	2957	2957
q14	308	268	289	268
q15	525	479	475	475
q16	483	406	408	406
q17	1809	1492	1502	1492
q18	7698	7561	7400	7400
q19	1708	1561	1604	1561
q20	1958	1812	1782	1782
q21	9640	5010	4996	4996
q22	1108	1025	1003	1003
Total cold run time: 76361 ms
Total hot run time: 54057 ms

@lsy3993
Copy link
Contributor Author

lsy3993 commented Aug 8, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39693 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 042462363ead863f034df700df52c850197cdf39, data reload: false

------ Round 1 ----------------------------------
q1	18466	4451	4289	4289
q2	2021	174	168	168
q3	10543	1155	1048	1048
q4	10145	768	724	724
q5	7519	2571	2534	2534
q6	227	142	138	138
q7	976	591	591	591
q8	9227	1904	1949	1904
q9	8695	6606	6555	6555
q10	7071	2216	2264	2216
q11	451	240	239	239
q12	394	228	216	216
q13	18755	2979	2943	2943
q14	294	235	241	235
q15	518	491	483	483
q16	521	390	389	389
q17	980	673	668	668
q18	8141	7434	7505	7434
q19	4119	1013	1110	1013
q20	681	313	336	313
q21	5267	4583	4619	4583
q22	1115	1045	1010	1010
Total cold run time: 116126 ms
Total hot run time: 39693 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4465	4242	4279	4242
q2	377	268	266	266
q3	2832	2631	2775	2631
q4	2005	1799	1740	1740
q5	5639	5529	5445	5445
q6	222	135	132	132
q7	2126	1770	1795	1770
q8	3296	3477	3524	3477
q9	8772	8775	8857	8775
q10	3476	3305	3179	3179
q11	598	486	487	486
q12	814	650	692	650
q13	16640	3161	3204	3161
q14	331	279	286	279
q15	537	495	515	495
q16	499	457	422	422
q17	1840	1529	1512	1512
q18	8191	8130	7879	7879
q19	1725	1632	1538	1538
q20	2174	1907	1871	1871
q21	8511	5436	5154	5154
q22	1198	1069	1012	1012
Total cold run time: 76268 ms
Total hot run time: 56116 ms

@lsy3993
Copy link
Contributor Author

lsy3993 commented Aug 14, 2024

run buildall

1 similar comment
@lsy3993
Copy link
Contributor Author

lsy3993 commented Aug 15, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38038 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 042462363ead863f034df700df52c850197cdf39, data reload: false

------ Round 1 ----------------------------------
q1	19419	5650	4413	4413
q2	2676	215	224	215
q3	10952	1256	1189	1189
q4	10510	670	825	670
q5	8092	2855	2857	2855
q6	274	155	155	155
q7	987	641	627	627
q8	9378	2163	2102	2102
q9	7280	6559	6534	6534
q10	7046	2243	2273	2243
q11	492	265	276	265
q12	417	244	247	244
q13	17777	3063	3011	3011
q14	306	257	250	250
q15	557	520	524	520
q16	522	403	399	399
q17	1012	642	723	642
q18	7481	6865	6679	6679
q19	5281	1061	981	981
q20	709	362	346	346
q21	3846	2777	2679	2679
q22	1124	1024	1019	1019
Total cold run time: 116138 ms
Total hot run time: 38038 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4600	4284	4292	4284
q2	412	300	297	297
q3	2837	2645	2618	2618
q4	1974	1650	1643	1643
q5	5360	5421	5377	5377
q6	232	143	143	143
q7	2097	1697	1738	1697
q8	3176	3374	3357	3357
q9	8430	8415	8359	8359
q10	3425	3188	3156	3156
q11	615	526	535	526
q12	805	652	652	652
q13	17303	2996	3026	2996
q14	313	300	288	288
q15	570	524	517	517
q16	496	438	453	438
q17	1792	1531	1471	1471
q18	7755	7577	7479	7479
q19	1740	1627	1662	1627
q20	2042	1821	1835	1821
q21	5405	5229	5194	5194
q22	1157	1043	1055	1043
Total cold run time: 72536 ms
Total hot run time: 54983 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190415 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 042462363ead863f034df700df52c850197cdf39, data reload: false

query1	993	406	390	390
query2	6727	1958	1899	1899
query3	6665	221	230	221
query4	33265	23658	23475	23475
query5	4397	629	630	629
query6	312	189	190	189
query7	4609	310	322	310
query8	408	361	360	360
query9	8591	2510	2482	2482
query10	478	341	325	325
query11	18019	15004	15245	15004
query12	172	123	124	123
query13	1668	414	410	410
query14	9776	7424	7786	7424
query15	329	191	190	190
query16	7446	549	536	536
query17	1835	597	580	580
query18	1318	327	344	327
query19	349	160	162	160
query20	142	133	129	129
query21	277	139	143	139
query22	4280	4010	4069	4010
query23	34375	33304	33204	33204
query24	11189	2905	2887	2887
query25	640	412	421	412
query26	1128	175	175	175
query27	2239	295	298	295
query28	6628	2127	2107	2107
query29	699	435	448	435
query30	343	189	178	178
query31	1058	847	837	837
query32	120	79	82	79
query33	807	340	333	333
query34	900	494	508	494
query35	925	762	787	762
query36	1118	950	934	934
query37	173	105	102	102
query38	3912	3832	3855	3832
query39	1519	1469	1495	1469
query40	243	155	152	152
query41	136	135	133	133
query42	134	116	117	116
query43	536	508	531	508
query44	1282	779	776	776
query45	221	193	194	193
query46	1130	779	775	775
query47	1905	1817	1841	1817
query48	408	341	348	341
query49	1194	577	600	577
query50	866	460	464	460
query51	6907	6796	6636	6636
query52	116	105	108	105
query53	296	227	229	227
query54	1010	513	502	502
query55	89	87	87	87
query56	330	304	307	304
query57	1262	1114	1114	1114
query58	332	306	311	306
query59	2987	2661	2796	2661
query60	354	333	324	324
query61	152	148	143	143
query62	875	694	697	694
query63	257	223	224	223
query64	5196	2378	1880	1880
query65	3230	3166	3188	3166
query66	1097	673	667	667
query67	15610	15018	14759	14759
query68	6613	580	582	580
query69	727	426	330	330
query70	1264	1200	1175	1175
query71	560	314	311	311
query72	7555	2310	2101	2101
query73	856	356	360	356
query74	9218	8722	8810	8722
query75	4844	2771	2766	2766
query76	4818	1072	1054	1054
query77	902	436	430	430
query78	9915	9115	11583	9115
query79	4973	552	556	552
query80	1122	610	596	596
query81	631	258	263	258
query82	408	159	155	155
query83	399	210	211	210
query84	291	96	103	96
query85	1522	355	340	340
query86	452	309	334	309
query87	4467	4430	4175	4175
query88	3755	2515	2499	2499
query89	439	327	327	327
query90	2096	224	219	219
query91	152	125	123	123
query92	85	74	75	74
query93	1145	558	554	554
query94	1028	321	326	321
query95	392	289	288	288
query96	595	281	283	281
query97	3271	3074	3070	3070
query98	246	221	238	221
query99	1634	1330	1314	1314
Total cold run time: 304359 ms
Total hot run time: 190415 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.18 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 042462363ead863f034df700df52c850197cdf39, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.05	0.05
query3	0.23	0.05	0.05
query4	1.66	0.08	0.08
query5	0.50	0.48	0.49
query6	1.13	0.73	0.74
query7	0.03	0.02	0.02
query8	0.06	0.06	0.05
query9	0.54	0.49	0.50
query10	0.54	0.55	0.55
query11	0.16	0.13	0.12
query12	0.15	0.13	0.14
query13	0.62	0.62	0.60
query14	0.77	0.82	0.78
query15	0.86	0.83	0.86
query16	0.36	0.38	0.38
query17	1.03	1.05	0.95
query18	0.23	0.22	0.23
query19	1.92	1.88	1.87
query20	0.02	0.01	0.01
query21	15.40	0.82	0.65
query22	4.45	7.89	1.47
query23	18.24	1.30	1.26
query24	2.11	0.24	0.22
query25	0.16	0.08	0.09
query26	0.30	0.21	0.22
query27	0.46	0.23	0.23
query28	13.28	1.04	1.00
query29	12.64	3.56	3.58
query30	0.43	0.24	0.24
query31	2.81	0.40	0.39
query32	3.25	0.49	0.48
query33	2.99	2.97	2.97
query34	17.25	4.34	4.38
query35	4.47	4.44	4.44
query36	0.68	0.48	0.51
query37	0.21	0.17	0.18
query38	0.17	0.17	0.17
query39	0.07	0.06	0.06
query40	0.18	0.15	0.14
query41	0.11	0.07	0.08
query42	0.08	0.06	0.07
query43	0.07	0.06	0.08
Total cold run time: 110.75 s
Total hot run time: 31.18 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 16, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit d189b8a into apache:master Aug 16, 2024
31 of 32 checks passed
dataroaring pushed a commit that referenced this pull request Aug 17, 2024
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
Related PR is : #37031 , when replay rename table, addTableRename is
needed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants