add mysql_cdc into demo

[jingchen2222]

Background

Create a demo to backup mysql into db3.
Resolve dbpunk-labs/db3#86

Todo list

• add mysql_cdc
• filter and convert binlog event into SQL statements
• create ns with delegate and run sql statement

[jingchen2222]

Reference

• convert binlog event into sql statemen: https://github.com/danfengcao/binlog2sql/blob/master/binlog2sql/binlog2sql.py
• binlog configuration: No field name in TABLE_MAP_EVENT? shyiko/mysql-binlog-connector-java#24

[jingchen2222]

Build and run db3

git clone https://github.com/dbpunk-labs/db3.git
cd db3
cargo build --release
RUST_LOG=info ./target/release/db3 --dev

Run mysql and config binlog

mysql -u root -p
SHOW global variables like 'binlog_%';
SHOW variables like 'binlog_%';
set global binlog_format = 'STATEMENT';
set binlog_format = 'STATEMENT';
set global binlog_row_metadata = 'FULL';
set binlog_row_metadata = 'FULL';

Execute mysql command:

> create schema main;
> use main;
> create table t1(c1 int, c2 int, c3 int);
> insert into t1 values(1,2,3);
> insert into t1 values(2,3,4);
> insert into t1 values(3,4,5);
> insert into t1(c1,c2) values(4,5);
> update t1 set c3 = 6 where c1 = 4;
> insert into t1 values(5,6,7);
> update t1 set c3 = 100 where c1 = 4;
> create table t2(c1 int, c2 bigint, c3 VARCHAR(50), c4 TIMESTAMP, c5 DATETIME);
> insert into t1 values(7,8,9)
> insert into t2 values (1,2,"abc", "2022-10-17 10:20:30", "2022-10-17 20:30:40")
> insert into t2 values (1,2,'abc', '2022-10-17 10:20:30', '2022-10-17 20:30:40')

Run example_db3_mysql_backup.

1. Alice create ns and add Bob delegate.
2. Bob sync mysql binlog and run sql with delegate

RUST_LOG=warn cargo run --example example_db3_mysql_backup -- --ns demo_ns --username root --password ${YOUR_MYSQL_PASSWROD}

LOGs:

[+] Subscribe to events ...
[+] Subscribe to events ...
[+] Account Nonce is 0

[+] req_id: 1, create ns >>>>>>>>
[+] Transaction got included in block Some(0xa2993d5b3e2379cb4408e5a6014cd13d97eb03c3c48da11969905b4af3268c36)
[+] GeneralResultEvent:
 {"status":0,"msg":"ok","req_id":"1"}
[+] Delegate Account Nonce is 0

[+] req_id: 1, runSqlByDelegate: create table t1(c1 int, c2 int, c3 int)
[+] Transaction got included in block Some(0x8690890f1cc0fa27ee19a3b0ed9dfe41e6bf4d45bd0ed2f660b87cb72a8103b4)
[+] GeneralResultEvent:
 {"status":0,"msg":"ok","req_id":"1"}
[2022-10-19T01:33:45Z WARN  example_db3_mysql_backup] Skip BEGIN statement
[+] Delegate Account Nonce is 1

[+] req_id: 2, runSqlByDelegate: insert into t1 values(1,2,3)
[+] Transaction got included in block Some(0x1a669ff1f7c549e5cbe462d3708bc7555c51862268e603fe66780d25a044ab65)
[2022-10-19T01:33:48Z WARN  example_db3_mysql_backup] ignoring event with unexpected req_id 1
[+] GeneralResultEvent:
 {"status":0, "msg":"ok","req_id":"2"}
[2022-10-19T01:33:51Z WARN  example_db3_mysql_backup] Skip BEGIN statement
[+] Delegate Account Nonce is 2

[+] req_id: 3, runSqlByDelegate: insert into t1 values(2,3,4)
[+] Transaction got included in block Some(0xd9cb3c759e3abb27d520a41d45bf9ef74c106fdceaaa7ef5df355af89e64cd9a)
[+] GeneralResultEvent:
 {"status":0, "msg":"ok","req_id":"3"}
[2022-10-19T01:33:57Z WARN  example_db3_mysql_backup] Skip BEGIN statement
[+] Delegate Account Nonce is 3

[+] req_id: 4, runSqlByDelegate: insert into t1 values(3,4,5)
[+] Transaction got included in block Some(0x45f9013a99bd4be0f0d5d8de4f95cb8289f7e9264a624ffde9fa71f82619d02b)
[+] GeneralResultEvent:
 {"status":0, "msg":"ok","req_id":"4"}
[2022-10-19T01:34:03Z WARN  example_db3_mysql_backup] Skip BEGIN statement
[+] Delegate Account Nonce is 4

[+] req_id: 5, runSqlByDelegate: insert into t1(c1,c2) values(4,5)
[+] Transaction got included in block Some(0x6674f3cf716638a126cf5b14fdc419d8d4d4f60f14d8373fedd986766b7d1cdf)
[+] GeneralResultEvent:
 {"status":0, "msg":"ok","req_id":"5"}
[2022-10-19T01:34:09Z WARN  example_db3_mysql_backup] Skip BEGIN statement
[+] Delegate Account Nonce is 5

[+] req_id: 6, runSqlByDelegate: update t1 set c3 = 6 where c1 = 4
[+] Transaction got included in block Some(0xb06ce6b7d0cbc6a639f165ab632a42becd1c857576d9b23879ce164ada18de51)
[+] GeneralResultEvent:
 {"status":0, "msg":"ok","req_id":"6"}
[2022-10-19T01:34:15Z WARN  example_db3_mysql_backup] Skip BEGIN statement
[+] Delegate Account Nonce is 6

[+] req_id: 7, runSqlByDelegate: insert into t1 values(5,6,7)
[+] Transaction got included in block Some(0x86fb39da648b7ec97809df5165fef0fc5bdc95911b6f9c5c959be5521974ce57)
[+] GeneralResultEvent:
 {"status":0, "msg":"ok","req_id":"7"}
[2022-10-19T01:34:21Z WARN  example_db3_mysql_backup] Skip BEGIN statement
[+] Delegate Account Nonce is 7

[+] req_id: 8, runSqlByDelegate: update t1 set c3 = 100 where c1 = 4
[+] Transaction got included in block Some(0x035da0511d1946869955ac698cea76c15c473a04566b94338bb2bc919299c076)
[+] GeneralResultEvent:
 {"status":0, "msg":"ok","req_id":"8"}
[2022-10-19T01:34:27Z WARN  example_db3_mysql_backup] Skip BEGIN statement
[+] Delegate Account Nonce is 8

[+] req_id: 9, runSqlByDelegate: insert into t1 values(7,8,9)
[+] Transaction got included in block Some(0x45213659100a3a573f8af23cf27ab93ae1b428502b4dddc0299ccecbac96a693)
[+] GeneralResultEvent:
 {"status":0, "msg":"ok","req_id":"9"}
[+] Delegate Account Nonce is 9

[+] req_id: 10, runSqlByDelegate: create table t2(c1 int, c2 bigint, c3 VARCHAR(50), c4 TIMESTAMP, c5 DATETIME)
[+] Transaction got included in block Some(0x22693202204cb00973ecbf7efd5072131b830705f0ad885c9826e4ea958dc874)
[+] GeneralResultEvent:
 {"status":0, "msg":"ok","req_id":"10"}
[2022-10-19T01:35:37Z WARN  example_db3_mysql_backup] Skip BEGIN statement
[+] Delegate Account Nonce is 10

[+] req_id: 11, runSqlByDelegate: insert into t2 values (1,2,'abc', '2022-10-17 10:20:30', '2022-10-17 20:30:40')
[+] Transaction got included in block Some(0x589d5e89f27243d01b475a8cc61e3d15278ebd5cb18ab36efca062677f3b31bd)
[+] GeneralResultEvent:
 {"status":0, "msg":"ok","req_id":"11"}
[2022-10-19T01:35:49Z WARN  example_db3_mysql_backup] Skip BEGIN statement
[+] Delegate Account Nonce is 11

[+] req_id: 12, runSqlByDelegate: insert into t2 values (1,2,'abc', '2022-10-17 10:20:30', '2022-10-17 20:30:40')
[+] Transaction got included in block Some(0x15af0260a72dc30b77c91a71461413e45f0744971d321c6ffd0c7bf8029cc807)
[+] GeneralResultEvent:
 {"status":0, "msg":"ok","req_id":"12"}

Check db3 data:

[jingchen2222]

Future work

• optimized workflow: batch run sql statements
• support binlog row mode
• support multi database. (can only support "main" database currently)
• support more column type. (e.g VARCHAR, TIMESTAMP, DATETIME, DATE, BLOB, etc)

[jingchen2222]

Support insert colume timestamp, datetime, varchar DONE