Skip to content

Commit cd4581b

Browse files
Pushapglinikep
Pushapgl
authored and
inikep
committed
[raft] Support for dumping raft logs to vanilla async replicas
Summary: [Porting Notes] We want to dump raft logs to vanilla async replicas regardless of whether it's the relay log or binlog. Effectively after this change we'll dump relay logs on the followers and binlogs on the leader. When the raft role changes, the logs to the dumped are also changed. Dump_log class is introduced as a thin wrapper/continer around mysql_bin_log or rli->relay_log and is inited with mysql_bin_log to emulate vanilla mysql behavior. Dump threads use the global dump_log object instead of mysql_bin_log directly. We switch the log in dump log only when raft role changes (in binlog_change_to_binlog() and binlog_change_to_apply_log()). During raft role change we take all log releated locks (LOCK_log, LOCK_index, LOCK_binlog_end_pos, and dump log lock) to serialize it with other log operations like dumping logs. Related doc - https://fb.quip.com/oTVAAdgEi4zY This diff contains below 7 patches: D23013977 D24766787 D24716539 D24900223 D24955284 D25174166 D25775525 Reviewed By: luqun Differential Revision: D26141496 ------------------------------------------------------------------------------- Passing raw_log pointer to wait_with_heartbeat() and wait_without_heartbeat() Summary: When enable_raft_plugin is OFF Dump_log::lock() is a no-op. Which means that when enable_raft_plugin is OFF there can be a race between log switching and dump threads. This could lead to a scenario where the raw_log that wait_next_event() is working on might be different than what wait_with_heartbeat()/wait_without_heartbeat() is working on. This can cause deadlocks because wait_with_heartbeat()/wait_without_heartbeat()'s mysql_cond_wait would unlock and then lock a different log's LOCK_binlog_end_pos mutex which would then never be unlocked by wait_next_event(). Reviewed By: anirbanr-fb Differential Revision: D32152658 fbshipit-source-id: d96ebcef966 ----------------------------------------------------------------------------------------- Fix rpl_raft_dump_raft_logs Summary: This tests completes but fails because the following warning exists: ``` 2022-08-30T16:28:00.159525Z 11 [ERROR] [MY-013114] [Repl] Slave I/O for channel '': Got fatal error 1236 from master when reading data from binary log: 'Slave has more GTIDs than the master has, using the master's SERVER_UUID. This may indicate that the end of the binary log was truncated or that the last binary log file was lost, e.g., after a power or disk failure when sync_binlog != 1. The master may or may not have rolled back transactions that were already replicated to the slave. Suggest to replicate any transactions that master has rolled back from slave to master, and/or commit empty transactions on master to account for transactions that have been', Error_code: MY-013114 ``` Since the MTR result file is valid, we can suppress this error. Reviewed By: yichenshen Differential Revision: D39141846 fbshipit-source-id: 8e7fdb8 ------------------------------------------------------------------------------- Fix heap overflow in group_relay_log_name handling Summary: We were accessing group_relay_log_name in Query_log_event::do_apply_event_worker() but it's assigned only after the coordinator thread encounters an end event (i.e. xid event or a query event with "COMMIT" or "ROLLBACK" query). This was causing a race between accessing group_relay_log_name in the worker thread and writing it on the coordinator thread. We don't need to set transaction position in events other than end event, so now we set transaction position in query event only if it's an end event. The race is eliminated because group_relay_log_name is set before enqueuing the event to the worker thread (in both dep repl and vanilla mts). Reviewed By: lth Differential Revision: D28767430 ------------------------------------------------------------------------------- fix memory during MYSQL_BIN_LOG::open_existing_binlog Summary: asandebug complain there are memory leaks during MYSQL_BIN_LOG open Direct leak of 50 byte(s) in 1 object(s) allocated from: #0 0x67460ef in malloc #1 0x93f0777 in my_raw_malloc(unsigned long, int) #2 0x93f064a in my_malloc(unsigned int, unsigned long, int) #3 0x93f0eb0 in my_strdup(unsigned int, char const*, int) #4 0x8af01a6 in MYSQL_BIN_LOG::open(unsigned int, char const*, char const*, unsigned int) #5 0x8af8064 in MYSQL_BIN_LOG::open_binlog(char const*, char const*, unsigned long, bool, bool, bool, Format_description_log_event*, unsigned int, RaftRotateInfo*, bool) #6 0x8b00c00 in MYSQL_BIN_LOG::new_file_impl(bool, Format_description_log_event*, RaftRotateInfo*) #7 0x8d65e47 in rotate_relay_log(Master_info*, bool, bool, bool, RaftRotateInfo*) #8 0x8d661c0 in rotate_relay_log_for_raft(RaftRotateInfo*) #9 0x8c7696a in process_raft_queue #10 0xa0fa1fd in pfs_spawn_thread(void*) #11 0x7f8c9a12b20b in start_thread release these memory before assign them Reviewed By: Pushapgl Differential Revision: D28819752
1 parent 9aa41b5 commit cd4581b

16 files changed

+1076
-98
lines changed

‎mysql-test/suite/rpl_raft/r/rpl_raft_dump_raft_logs.result

+288
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,288 @@
1+
include/raft_3_node.inc
2+
Warnings:
3+
Note #### Sending passwords in plain text without SSL/TLS is extremely insecure.
4+
Note #### Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
5+
Warnings:
6+
Note #### Sending passwords in plain text without SSL/TLS is extremely insecure.
7+
Note #### Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
8+
[connection master]
9+
include/rpl_connect.inc [creating server_4]
10+
include/rpl_connect.inc [creating server_5]
11+
RESET MASTER;
12+
RESET SLAVE;
13+
SET @@GLOBAL.ENABLE_RAFT_PLUGIN = 0;
14+
CHANGE MASTER TO MASTER_HOST = '::1', MASTER_PORT = SERVER_MYPORT_1, MASTER_USER = 'root', MASTER_CONNECT_RETRY = 1, MASTER_AUTO_POSITION = 1;
15+
Warnings:
16+
Note 1759 Sending passwords in plain text without SSL/TLS is extremely insecure.
17+
Note 1760 Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
18+
START SLAVE;
19+
RESET MASTER;
20+
RESET SLAVE;
21+
SET @@GLOBAL.ENABLE_RAFT_PLUGIN = 0;
22+
CHANGE MASTER TO MASTER_HOST = '::1', MASTER_PORT = SERVER_MYPORT_2, MASTER_USER = 'root', MASTER_CONNECT_RETRY = 1, MASTER_AUTO_POSITION = 1;
23+
Warnings:
24+
Note 1759 Sending passwords in plain text without SSL/TLS is extremely insecure.
25+
Note 1760 Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
26+
START SLAVE;
27+
show status like 'rpl_raft_role';
28+
Variable_name Value
29+
Rpl_raft_role LEADER
30+
show status like 'rpl_raft_role';
31+
Variable_name Value
32+
Rpl_raft_role FOLLOWER
33+
show status like 'rpl_raft_role';
34+
Variable_name Value
35+
Rpl_raft_role FOLLOWER
36+
show status like 'rpl_raft_role';
37+
Variable_name Value
38+
Rpl_raft_role
39+
show status like 'rpl_raft_role';
40+
Variable_name Value
41+
Rpl_raft_role
42+
create table t1 (a int primary key) engine = innodb;
43+
include/sync_slave_sql_with_master.inc
44+
include/sync_slave_sql_with_master.inc
45+
include/sync_slave_sql_with_master.inc
46+
include/sync_slave_sql_with_master.inc
47+
SELECT "LOGNAME" LIKE "binary-logs%";
48+
"LOGNAME" LIKE "binary-logs%"
49+
1
50+
SELECT "LOGNAME" LIKE "binary-logs%";
51+
"LOGNAME" LIKE "binary-logs%"
52+
1
53+
insert into t1 values(1);
54+
include/sync_slave_sql_with_master.inc
55+
include/sync_slave_sql_with_master.inc
56+
include/sync_slave_sql_with_master.inc
57+
include/sync_slave_sql_with_master.inc
58+
select * from t1;
59+
a
60+
1
61+
select * from t1;
62+
a
63+
1
64+
select * from t1;
65+
a
66+
1
67+
select * from t1;
68+
a
69+
1
70+
include/raft_promote_to_leader.inc
71+
insert into t1 values(2);
72+
insert into t1 values(3);
73+
insert into t1 values(4);
74+
include/sync_slave_sql_with_master.inc
75+
include/sync_slave_sql_with_master.inc
76+
include/sync_slave_sql_with_master.inc
77+
include/sync_slave_sql_with_master.inc
78+
select * from t1;
79+
a
80+
1
81+
2
82+
3
83+
4
84+
select * from t1;
85+
a
86+
1
87+
2
88+
3
89+
4
90+
select * from t1;
91+
a
92+
1
93+
2
94+
3
95+
4
96+
select * from t1;
97+
a
98+
1
99+
2
100+
3
101+
4
102+
SELECT "LOGNAME" LIKE "binary-logs%";
103+
"LOGNAME" LIKE "binary-logs%"
104+
1
105+
SELECT "LOGNAME" LIKE "binary-logs%";
106+
"LOGNAME" LIKE "binary-logs%"
107+
1
108+
include/raft_promote_to_leader.inc
109+
flush binary logs;
110+
insert into t1 values(5);
111+
include/sync_slave_sql_with_master.inc
112+
include/sync_slave_sql_with_master.inc
113+
include/sync_slave_sql_with_master.inc
114+
include/sync_slave_sql_with_master.inc
115+
select * from t1;
116+
a
117+
1
118+
2
119+
3
120+
4
121+
5
122+
select * from t1;
123+
a
124+
1
125+
2
126+
3
127+
4
128+
5
129+
select * from t1;
130+
a
131+
1
132+
2
133+
3
134+
4
135+
5
136+
select * from t1;
137+
a
138+
1
139+
2
140+
3
141+
4
142+
5
143+
set @@global.debug='+d,dump_wait_before_find_next_log';
144+
insert into t1 values(6);
145+
insert into t1 values(7);
146+
flush binary logs;
147+
insert into t1 values(8);
148+
insert into t1 values(9);
149+
flush binary logs;
150+
insert into t1 values(10);
151+
insert into t1 values(11);
152+
set debug_sync= 'now wait_for signal.reached';
153+
purge raft logs to 'LOGNAME';
154+
set debug_sync= 'now signal signal.done';
155+
set @@global.debug='-d,dump_wait_before_find_next_log';
156+
include/sync_slave_sql_with_master.inc
157+
include/sync_slave_sql_with_master.inc
158+
include/sync_slave_sql_with_master.inc
159+
include/sync_slave_sql_with_master.inc
160+
select * from t1;
161+
a
162+
1
163+
2
164+
3
165+
4
166+
5
167+
6
168+
7
169+
8
170+
9
171+
10
172+
11
173+
select * from t1;
174+
a
175+
1
176+
2
177+
3
178+
4
179+
5
180+
6
181+
7
182+
8
183+
9
184+
10
185+
11
186+
select * from t1;
187+
a
188+
1
189+
2
190+
3
191+
4
192+
5
193+
6
194+
7
195+
8
196+
9
197+
10
198+
11
199+
select * from t1;
200+
a
201+
1
202+
2
203+
3
204+
4
205+
5
206+
6
207+
7
208+
8
209+
9
210+
10
211+
11
212+
include/stop_slave.inc
213+
include/rpl_restart_server.inc [server_number=2]
214+
include/rpl_restart_server.inc [server_number=1]
215+
include/raft_promote_to_leader.inc
216+
START SLAVE IO_THREAD;
217+
START SLAVE IO_THREAD;
218+
insert into t1 values(12);
219+
include/sync_slave_sql_with_master.inc
220+
include/sync_slave_sql_with_master.inc
221+
include/sync_slave_sql_with_master.inc
222+
include/sync_slave_sql_with_master.inc
223+
select * from t1;
224+
a
225+
1
226+
2
227+
3
228+
4
229+
5
230+
6
231+
7
232+
8
233+
9
234+
10
235+
11
236+
12
237+
select * from t1;
238+
a
239+
1
240+
2
241+
3
242+
4
243+
5
244+
6
245+
7
246+
8
247+
9
248+
10
249+
11
250+
12
251+
select * from t1;
252+
a
253+
1
254+
2
255+
3
256+
4
257+
5
258+
6
259+
7
260+
8
261+
9
262+
10
263+
11
264+
12
265+
select * from t1;
266+
a
267+
1
268+
2
269+
3
270+
4
271+
5
272+
6
273+
7
274+
8
275+
9
276+
10
277+
11
278+
12
279+
drop table t1;
280+
include/sync_slave_sql_with_master.inc
281+
include/sync_slave_sql_with_master.inc
282+
include/sync_slave_sql_with_master.inc
283+
include/sync_slave_sql_with_master.inc
284+
stop slave;
285+
reset slave all;
286+
stop slave;
287+
reset slave all;
288+
include/rpl_end.inc

‎mysql-test/suite/rpl_raft/r/rpl_raft_purged_gtids_dump_threads.result

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
include/raft_3_node.inc
2+
Warnings:
3+
Note #### Sending passwords in plain text without SSL/TLS is extremely insecure.
4+
Note #### Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
5+
Warnings:
6+
Note #### Sending passwords in plain text without SSL/TLS is extremely insecure.
7+
Note #### Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
8+
[connection master]
9+
include/rpl_connect.inc [creating server_4]
10+
include/rpl_connect.inc [creating server_5]
11+
create table t1 (a int) engine = innodb;
12+
insert into t1 values(1);
13+
insert into t1 values(2);
14+
set @@global.rpl_raft_new_leader_uuid = 'uuid2';
15+
change master to master_host='localhost', master_port=port2, master_auto_position=1, master_user='root';
16+
Warnings:
17+
Note 1759 Sending passwords in plain text without SSL/TLS is extremely insecure.
18+
Note 1760 Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
19+
include/start_slave.inc
20+
insert into t1 values(3);
21+
include/sync_slave_sql_with_master.inc
22+
select * from t1;
23+
a
24+
1
25+
2
26+
3
27+
set @@global.rpl_raft_new_leader_uuid = 'uuid1';
28+
drop table t1;
29+
include/sync_slave_sql_with_master.inc
30+
include/sync_slave_sql_with_master.inc
31+
include/sync_slave_sql_with_master.inc
32+
include/stop_slave.inc
33+
reset slave all;
34+
include/rpl_end.inc

0 commit comments

Comments
 (0)