Skip to content

branch-3.0: [Fix](cloud) Cloud enable fe deploy mode from master-observers to multi-followers #45255 #47215

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
gavinchou merged 1 commit into from
Jan 23, 2025
Merged

branch-3.0: [Fix](cloud) Cloud enable fe deploy mode from master-observers to multi-followers #45255 #47215

gavinchou merged 1 commit into from
Jan 23, 2025

Conversation

@deardeng
Copy link
Contributor

@deardeng deardeng commented Jan 20, 2025

cherry pick from #45255

@deardeng
[Fix](cloud) Cloud enable fe deploy mode from master-observers to mul…
87e3470 
…ti-followers (apache#45255)

IN CLOUD
1. add some cloud metaservice api regression test
2. prohibit one node add to multi cluster repeatedly
3. supports switching from master observer mode to multi follower mode,
and compatible with upgrading from existing master-observers deployment
mode to multi-follwers mode
4. when adding an FE node(add fe cluster, add fe node), there is a
5-minute protection time and the FE node cannot be droped(drop fe
cluster, drop fe node)
5. drop the node. If the node cannot be found, an error will occur and
the code will return 404 instead of the previous silent processing
6. If the MS node starts with 127.0.0.1 IP and the
`prohibit_use_loopback_addresses` is set to true(default), the MS node
will not be able to start, ms log will find `enable check prohibit use
loopback addr, but localhost=127.0.0.1, so exit(-1)`
@deardeng deardeng requested a review from dataroaring as a code owner January 20, 2025 02:32
@deardeng
Copy link
Contributor Author

deardeng commented Jan 20, 2025

run buildall

@Thearas
Copy link
Contributor

Thearas commented Jan 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@deardeng
Copy link
Contributor Author

deardeng commented Jan 20, 2025

run feut

@doris-robot
Copy link

doris-robot commented Jan 20, 2025

TPC-H: Total hot run time: 40855 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 87e3470461aa503f3ef65e5da6b3e863a0951014, data reload: false

------ Round 1 ----------------------------------
q1	17923	7738	7239	7239
q2	2098	176	167	167
q3	10747	1068	1186	1068
q4	10536	731	723	723
q5	7740	2834	2852	2834
q6	238	149	148	148
q7	973	616	605	605
q8	9360	1962	2014	1962
q9	6697	6444	6438	6438
q10	7039	2363	2346	2346
q11	462	272	269	269
q12	409	216	207	207
q13	17800	2988	3011	2988
q14	237	204	219	204
q15	574	516	509	509
q16	687	613	619	613
q17	990	582	616	582
q18	7219	6879	6578	6578
q19	1390	1076	1068	1068
q20	481	202	198	198
q21	4073	3129	3169	3129
q22	1103	1023	980	980
Total cold run time: 108776 ms
Total hot run time: 40855 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7346	7200	7254	7200
q2	327	248	233	233
q3	2969	2955	2938	2938
q4	2125	1905	1783	1783
q5	5752	5750	5748	5748
q6	231	139	141	139
q7	2248	1883	1814	1814
q8	3340	3559	3510	3510
q9	8824	8907	8815	8815
q10	3560	3520	3572	3520
q11	615	499	506	499
q12	812	595	589	589
q13	9091	3177	3173	3173
q14	327	283	271	271
q15	605	535	537	535
q16	729	678	670	670
q17	1837	1644	1597	1597
q18	8155	7615	7586	7586
q19	1681	1560	1461	1461
q20	2124	1880	1858	1858
q21	5618	5335	5359	5335
q22	1169	1025	1033	1025
Total cold run time: 69485 ms
Total hot run time: 60299 ms

@doris-robot
Copy link

doris-robot commented Jan 20, 2025

TPC-DS: Total hot run time: 199273 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 87e3470461aa503f3ef65e5da6b3e863a0951014, data reload: false

query1	1331	954	928	928
query2	6277	2069	2049	2049
query3	10826	4492	4129	4129
query4	66356	29585	23455	23455
query5	4956	472	468	468
query6	425	196	188	188
query7	5682	317	316	316
query8	318	224	221	221
query9	9114	2680	2684	2680
query10	468	268	265	265
query11	17610	15419	15847	15419
query12	159	107	106	106
query13	1537	451	432	432
query14	10650	7840	7573	7573
query15	206	189	196	189
query16	7078	489	485	485
query17	1098	612	599	599
query18	1981	331	321	321
query19	235	163	173	163
query20	118	114	115	114
query21	207	109	106	106
query22	4594	4539	4392	4392
query23	34933	35303	34670	34670
query24	6185	3038	3018	3018
query25	546	433	425	425
query26	684	178	177	177
query27	1866	374	381	374
query28	4012	2519	2467	2467
query29	693	452	461	452
query30	256	171	167	167
query31	1007	841	853	841
query32	70	61	59	59
query33	429	282	284	282
query34	915	518	537	518
query35	858	767	762	762
query36	1117	956	994	956
query37	125	77	75	75
query38	4156	4024	4044	4024
query39	1544	1465	1512	1465
query40	209	102	99	99
query41	48	49	51	49
query42	119	105	103	103
query43	552	507	491	491
query44	1233	816	835	816
query45	187	176	169	169
query46	1175	758	783	758
query47	2060	1911	1926	1911
query48	493	379	381	379
query49	733	402	391	391
query50	863	434	434	434
query51	7312	7380	7030	7030
query52	100	89	87	87
query53	273	183	190	183
query54	585	465	467	465
query55	79	80	74	74
query56	257	255	246	246
query57	1235	1108	1098	1098
query58	220	201	214	201
query59	3234	3091	3103	3091
query60	282	252	256	252
query61	125	105	108	105
query62	778	675	660	660
query63	223	197	196	196
query64	1362	667	677	667
query65	3284	3236	3228	3228
query66	692	303	308	303
query67	16131	15594	15553	15553
query68	3967	584	582	582
query69	435	278	273	273
query70	1176	1123	1106	1106
query71	361	264	263	263
query72	6084	4059	3980	3980
query73	770	349	353	349
query74	10223	9041	8988	8988
query75	3401	2662	2694	2662
query76	1932	1086	1167	1086
query77	500	281	277	277
query78	10686	9699	9651	9651
query79	1541	603	615	603
query80	891	435	446	435
query81	509	244	240	240
query82	1289	121	115	115
query83	258	147	146	146
query84	286	88	76	76
query85	904	302	297	297
query86	348	305	296	296
query87	4562	4352	4415	4352
query88	3746	2396	2356	2356
query89	424	298	292	292
query90	1987	192	194	192
query91	182	150	148	148
query92	70	49	51	49
query93	1994	555	562	555
query94	813	310	315	310
query95	366	264	255	255
query96	626	275	299	275
query97	3356	3208	3212	3208
query98	218	199	201	199
query99	1614	1331	1288	1288
Total cold run time: 319985 ms
Total hot run time: 199273 ms

@deardeng
Copy link
Contributor Author

deardeng commented Jan 21, 2025

run feut

@deardeng
Copy link
Contributor Author

deardeng commented Jan 21, 2025

run cloud_p0

@gavinchou gavinchou merged commit c4c526a into apache:branch-3.0 Jan 23, 2025
20 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring Awaiting requested review from dataroaring

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@deardeng @Thearas @doris-robot @gavinchou