Skip to content

branch-3.0-pick: [Fix](cloud-mow) Fix race between CloudMetaMgr::sync_tablet_rowsets and CloudSchemaChangeJob::_convert_historical_rowsets (#50051) #50081

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Apr 22, 2025
Merged

branch-3.0-pick: [Fix](cloud-mow) Fix race between CloudMetaMgr::sync_tablet_rowsets and CloudSchemaChangeJob::_convert_historical_rowsets (#50051) #50081

dataroaring merged 1 commit into from
Apr 22, 2025

Conversation

@bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Apr 16, 2025

pick #50051

@bobhan1 bobhan1 requested a review from dataroaring as a code owner April 16, 2025 03:05
@Thearas
Copy link
Contributor

Thearas commented Apr 16, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1
[Fix](cloud-mow) Fix race between CloudMetaMgr::sync_tablet_rowsets…
76cec48 
… and `CloudSchemaChangeJob::_convert_historical_rowsets` (apache#50051)

Considering the following race situation:
1. thread 1 finish convert historical rowsets before version Y on tablet
A and write tmp converted rowset metas on MS
2. thread 2 begins sync_rowsets on tablet A with version Y and get
visible rowsets before version Y.
3. thread 1 commit heavy schema change job on MS and turn tmp converted
historical rowsets to visible rowsets on tablet Y.
4. thread 1 add converted historical rowset metas to tablet Y's BE local
tablet meta and remove all delete bitmaps of new tablet before version
Y.
5. thread 2 **add rowsets to tablet Y's BE local tablet meta which
overwrites schema change's converted rowsets**.

This will cause correctness problem. This PR add a lock to avoid this
situation.
@bobhan1 bobhan1 force-pushed the branch-3.0-pick-50051 branch from 6ea3266 to 76cec48 Compare April 16, 2025 03:06
@bobhan1
Copy link
Contributor Author

bobhan1 commented Apr 16, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 16, 2025

TPC-H: Total hot run time: 40369 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 76cec4898fbfa85449d29e7282cec7d8bd15d5e9, data reload: false

------ Round 1 ----------------------------------
q1	17574	7353	6658	6658
q2	2055	180	180	180
q3	10615	1070	1176	1070
q4	10552	764	730	730
q5	7732	2894	2848	2848
q6	217	135	137	135
q7	976	630	615	615
q8	9363	1940	2016	1940
q9	6547	6382	6420	6382
q10	7026	2249	2261	2249
q11	459	261	268	261
q12	410	223	221	221
q13	17770	3006	3012	3006
q14	244	208	213	208
q15	500	456	471	456
q16	650	591	584	584
q17	997	599	569	569
q18	7171	6726	6684	6684
q19	1409	1127	1070	1070
q20	476	206	211	206
q21	4082	3395	3317	3317
q22	1136	980	1006	980
Total cold run time: 107961 ms
Total hot run time: 40369 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6705	6676	6691	6676
q2	330	243	234	234
q3	2900	2782	2986	2782
q4	2028	1850	1823	1823
q5	5730	5761	5716	5716
q6	211	128	131	128
q7	2204	1837	1834	1834
q8	3418	3537	3554	3537
q9	8794	8901	8848	8848
q10	3596	3544	3538	3538
q11	585	488	493	488
q12	832	597	624	597
q13	8371	3199	3169	3169
q14	293	293	276	276
q15	506	468	468	468
q16	693	660	663	660
q17	1862	1625	1626	1625
q18	8350	7607	7652	7607
q19	1677	1542	1492	1492
q20	2123	1871	1836	1836
q21	5551	5582	5298	5298
q22	1131	1062	1046	1046
Total cold run time: 67890 ms
Total hot run time: 59678 ms

@doris-robot
Copy link

doris-robot commented Apr 16, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 40.03% (10489/26200)
Line Coverage 30.71% (88167/287078)
Region Coverage 29.82% (45441/152370)
Branch Coverage 26.20% (23036/87936)

@doris-robot
Copy link

doris-robot commented Apr 16, 2025

TPC-DS: Total hot run time: 197961 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 76cec4898fbfa85449d29e7282cec7d8bd15d5e9, data reload: false

query1	1315	954	916	916
query2	6250	2074	1986	1986
query3	10984	4544	4573	4544
query4	32895	23491	23894	23491
query5	4691	469	464	464
query6	275	182	182	182
query7	4003	319	326	319
query8	275	223	216	216
query9	9595	2596	2589	2589
query10	500	283	264	264
query11	17758	15220	15135	15135
query12	160	103	105	103
query13	1560	445	423	423
query14	9746	7086	7192	7086
query15	229	193	189	189
query16	8049	501	485	485
query17	1614	628	613	613
query18	2127	339	330	330
query19	253	175	177	175
query20	133	126	122	122
query21	221	106	110	106
query22	4663	4450	4467	4450
query23	34928	33731	34018	33731
query24	11142	2993	2979	2979
query25	645	411	402	402
query26	1178	178	174	174
query27	2378	355	353	353
query28	7332	2446	2448	2446
query29	853	461	449	449
query30	278	171	165	165
query31	1022	821	848	821
query32	100	56	57	56
query33	794	306	311	306
query34	1215	516	522	516
query35	875	744	753	744
query36	1116	975	985	975
query37	140	69	66	66
query38	4096	4011	4035	4011
query39	1516	1465	1454	1454
query40	202	101	102	101
query41	50	48	50	48
query42	119	107	100	100
query43	554	516	501	501
query44	1341	842	828	828
query45	191	169	173	169
query46	1165	740	753	740
query47	2047	1930	1959	1930
query48	493	404	412	404
query49	987	423	412	412
query50	852	434	441	434
query51	7400	7260	7129	7129
query52	110	91	92	91
query53	278	200	190	190
query54	1120	486	483	483
query55	86	79	81	79
query56	265	264	260	260
query57	1196	1094	1092	1092
query58	246	222	217	217
query59	3132	2998	2876	2876
query60	294	277	281	277
query61	130	128	147	128
query62	853	694	685	685
query63	228	211	196	196
query64	4130	771	670	670
query65	3290	3208	3229	3208
query66	840	307	299	299
query67	15781	15638	15619	15619
query68	4468	596	591	591
query69	445	282	288	282
query70	1243	1119	1142	1119
query71	344	259	266	259
query72	6264	4096	4020	4020
query73	768	346	359	346
query74	10152	9193	9248	9193
query75	3390	2651	2654	2651
query76	2784	1083	1114	1083
query77	435	288	266	266
query78	10583	9608	9647	9608
query79	1630	617	605	605
query80	909	453	436	436
query81	571	246	245	245
query82	586	95	93	93
query83	247	153	141	141
query84	240	88	78	78
query85	1688	311	301	301
query86	489	312	297	297
query87	4371	4265	4253	4253
query88	4265	2391	2377	2377
query89	413	299	303	299
query90	2044	187	188	187
query91	189	168	151	151
query92	64	50	55	50
query93	2315	564	560	560
query94	922	305	291	291
query95	360	259	263	259
query96	621	282	287	282
query97	3279	3161	3161	3161
query98	222	209	194	194
query99	1503	1303	1281	1281
Total cold run time: 302088 ms
Total hot run time: 197961 ms

@doris-robot
Copy link

doris-robot commented Apr 16, 2025

ClickBench: Total hot run time: 31.96 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 76cec4898fbfa85449d29e7282cec7d8bd15d5e9, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.07
query4	1.62	0.10	0.11
query5	0.53	0.49	0.53
query6	1.14	0.72	0.74
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.56	0.52	0.50
query10	0.55	0.54	0.56
query11	0.15	0.12	0.11
query12	0.14	0.10	0.11
query13	0.61	0.60	0.60
query14	2.73	2.73	2.87
query15	0.90	0.82	0.82
query16	0.37	0.39	0.38
query17	1.05	1.06	1.04
query18	0.23	0.21	0.22
query19	1.95	1.81	2.01
query20	0.01	0.02	0.01
query21	15.36	0.57	0.56
query22	2.54	1.83	1.43
query23	17.07	0.87	0.86
query24	3.10	1.18	1.07
query25	0.28	0.27	0.11
query26	0.32	0.15	0.13
query27	0.04	0.03	0.03
query28	10.20	0.49	0.52
query29	12.55	3.24	3.23
query30	0.25	0.06	0.06
query31	2.86	0.38	0.37
query32	3.26	0.48	0.46
query33	2.97	3.04	3.00
query34	17.27	4.56	4.55
query35	4.57	4.53	4.51
query36	0.68	0.48	0.48
query37	0.10	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.03
query42	0.03	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.74 s
Total hot run time: 31.96 s

@bobhan1
Copy link
Contributor Author

bobhan1 commented Apr 16, 2025

run p0

dataroaring
dataroaring approved these changes Apr 22, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 750da9f into apache:branch-3.0 Apr 22, 2025
20 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bobhan1 @Thearas @doris-robot @dataroaring